Initial commit of BEMANI Utilities to GitHub.

.gitignore

@@ -0,0 +1,13 @@
+__pycache__
+*.swp
+*.swo
+*.pyc
+*.pyo
+.mypy_cache/
+*.c
+*.o
+*.so
+build/
+.hg/
+.hgignore
+.venv/

2dxutils

@@ -0,0 +1,4 @@
+#! /bin/bash
+
+export PYTHONPATH=$(python -c "import os; print(os.path.realpath('.'))")
+python3 -m bemani.utils.twodxutils "$@"

BACKEND.md

@@ -0,0 +1,98 @@
+The backend services component is responsible for parsing game requests
+and generating an appropriate response. It is divided into a dispatch
+layer and a series of folders each representing a game series. There is
+also a data layer filled with database access functions. Each game
+series knows how to dispatch a request to an appropriate game class
+given a model string and the Node structure (as detailed in PROTOCOL).
+Each game class inherits from the base backend class which provides common
+functionality such as boot requests, PASELI, and card lookup routines.
+Both the base backend class and the game classes use the data layer to
+determine appropriate responses given a request.
+
+A rough sketch of how the pieces fit together is as follows:
+
+                      ------------
+                      | Dispatch |
+                      ------------
+                           |
+                           |
+                           V
+       --------     --------------     --------------
+       | Base |<--->| Game Class |<--->| Data Class |
+       --------     --------------     --------------
+
+Dispatch is responsible for taking a tree of Node objects and determining
+which game class should handle it. It will extract the modelstring and
+root Node's name in order to determine which game and what request is
+being made. It will extract the PCBID of the request and optionally compare
+it against known PCBIDs if enforcing mode is enabled. If the PCBID is tied
+to an arcade, it will look up the details of that arcade in order to allow
+an operator to override PASELI settings. Then, it create a database connection
+using the Data class and instantiates the appropriate game class with the
+Data class and modelstring provided on instantiation. Then, it attempts to
+call the `handle_<request>_<method>_request` method on the instantiated game
+class. The <request> portion represents the name of the root Node object, and
+the <method> portion represents the string method found as an attribute on the
+root node. If that doesn't exist, it falls back to the `handle_<request>_request`
+method. Some series prefer to handle everything in one request method, manually
+examining the method, and some prefer to have individual methods per request
+and method combination. If neither of these exists, then it is assumed that
+the game does not handle this packet and an appropriate error is generated.
+
+Under most circumstances, code has been copied forward to new games in a series
+instead of made common. This is because games tend to subtly change their
+expectations of the network protocol under the assumption that the player will
+never go back to older versions of the game or server. A conscious decision was
+made to copy-paste code instead of trying to refactor it to be common, specifically
+to represent the reality that while the code appears identical/similar enough,
+its really up to the game engineers how to implement and things are subtly different.
+In some cases, a packet truly doesn't change across versions and in that case
+game series will chose to factor out common code, but in many cases the choice was
+made to sacrifice ease of maintainability for stability across versions.
+
+All game classes should inherit from Base. It has several
+`handle_<request>_request` handlers for basic functions such as PASELI, bootup,
+and card lookup routines. Base assumes that `has_profile`, `get_profile`,
+`put_profile`, and `bind_profile` are subclassed by any game wishing to provide
+complex profile handling, but provides default handlers which should work for any
+simple game. It also provides `get_play_statistics` and `update_play_statistics`
+which all games should use during their respective profile fetch and save. Base
+handles looking up a user's PASELI balance if the PCBID of the request is in an
+arcade and that arcade is set to non-infinite PASELI.
+
+Each game class is expected to have a few properties. The 'game' property should
+be a string name and is used for all DB operations when referring to a game
+series. The 'version' property should be an integer and is used for all DB
+operations when referring to a specific version of a game. Game code is free to
+choose what these values are set to, but currently all games use constants
+defined in `bemani/common/constants.py` for easier sharing with the BEMAPI REST
+server and the frontend. This is how Base can provide statistics, profile lookup
+and card manager services without knowing anything about what game is subclassin
+g it. Aside from that, games are free to handle packets in any way they see fit.
+
+In general, the tables provided by the database are indexed document stores.
+Most BEMANI games are implemented entirely client-side and simply expect the
+server to return the data it sent to it on the last profile save. For that to
+work, there are a few database abstractions in use across various games. The
+'profile' table stores JSON data given a specific game, so it is appropriate
+for game-specific data. The 'game_settings' table stores JSON data given
+a game series, so it is appropriate for data that is consistent across an
+entire series. Note that this does not include scores! The 'achievement' table
+stores JSON data given a specific game, an identifier and an identifier type.
+This table is perfect for item data such as unlocks or event progress. The
+'score' table stores JSON given a game and song ID, as does the 'score_history'
+table. Note that the song ID is an internal representation and translated
+through the 'music' table. This is to allow game series to renumber their
+songs while still keeping score history across versions. It also allows the
+server to preserve scores across multiple versions of a game. Note that all above
+tables are accessed through the Data class instead of directly creating SQL
+in the game classes.
+
+In some cases, we access a remote version of the Data classes instead of the
+MySQL version directly. The remote version in turn contacts any remote BEMAPI
+REST servers as well as the local database and then sums up the information before
+returning it to the game layer. In this way, we support fetching scores and
+rivaling across networks using the BEMAPI REST API in a manner that is virtually
+transparent to individual game implementations. Crucially, it is not used in
+the API implementation itself nor in the frontend, ensuring that both only
+respond with data that is contained on this instance directly.

MANIFEST.in

@@ -0,0 +1,25 @@
+include bemani/frontend/templates/*.html
+include bemani/frontend/templates/admin/*.html
+include bemani/frontend/templates/account/*.html
+include bemani/frontend/static/*.css
+include bemani/frontend/static/*.js
+include bemani/frontend/static/*.gif
+include bemani/frontend/static/*.png
+include bemani/frontend/static/components/*.js
+include bemani/frontend/static/controllers/*.js
+include bemani/frontend/static/controllers/admin/*.js
+include bemani/frontend/static/controllers/account/*.js
+include bemani/frontend/static/controllers/arcade/*.js
+include bemani/frontend/static/controllers/common/*.js
+include bemani/frontend/static/controllers/iidx/*.js
+include bemani/frontend/static/controllers/popn/*.js
+include bemani/frontend/static/controllers/jubeat/*.js
+include bemani/frontend/static/controllers/bishi/*.js
+include bemani/frontend/static/controllers/ddr/*.js
+include bemani/frontend/static/controllers/reflec/*.js
+include bemani/frontend/static/controllers/sdvx/*.js
+include bemani/frontend/static/controllers/museca/*.js
+exclude bemani/protocol/lz77.py
+exclude bemani/protocol/stream.py
+exclude bemani/protocol/binary.py
+exclude bemani/protocol/xml.py

PROTOCOL.md

@@ -0,0 +1,59 @@
+The eAmusement protocol layer is divided into the main encoder/decoder class,
+a class for parsing old-style XML, a class for parsing new-style binary tree
+structure, a class representing a single node in a tree, and a few helper
+classes to tie the whole system together. Each message as sent to or received
+from a game can be represented as a tree of nodes. A node can either have
+additional nodes as children, or it can have a data value. Both types of node
+can have attributes. Given a tree of nodes, the encoder will output valid
+binary data suitable for returning to a game over HTTP, including any optional
+encryption or compression. Given binary data posted over HTTP from a game, the
+decoder will output a tree of nodes.
+
+A rough sketch of how the pieces fit together is as follows:
+
+            ------------------             --------
+            | EAmuseProtocol |------------>| Lz77 |
+            ------------------             --------
+                 |      |
+                 |      -------------------
+                 |                        |
+                 V                        V
+          ---------------           ------------------
+          | XmlEncoding |           | BinaryEncoding |
+          ---------------           ------------------
+             |       ^                 ^         |
+             |       |   ----------    |         |
+             |       --->| Stream |<----         |
+             |           ----------              |
+             |                                   |
+             |            --------               |
+             ------------>| Node |<---------------
+                          --------
+
+A packet will come in as data representing XML or a binary packet. It is
+optionally wrapped with Lz77 compression. That is optionally wrapped with
+RC4 encryption. Note that a packet may be encrypted and not compressed, but
+a packet with both compression and encryption will have RC4 as the outermost
+layer, followed by Lz77, followed finally by the raw data either as XML or
+binary.
+
+EAmuseProtocol is responsible for encryption/decryption using inlined RC4
+code, Lz77 compression/decompression using the Lz77 helper class, and finally
+uses either the XmlEncoding or BinaryEncoding class to convert to/from a
+tree of Node objects. Both XmlEncoding and BinaryEncoding use the Stream class
+as a helper for creating and dissecting raw binary data that will be exchanged
+with EamuseProtocol. Finally, Node is a representation of one element in the
+tree, having a name, an optional value and optional children which are also
+instances of the Node class.
+
+This setup is designed from the perspective of having a HTTP server such as
+flask pass binary data to EAmuseProtocol, and retrieve encoded responses from
+it. Game server code is expected to receive a tree in the form of a root Node
+instance. It will use various helper methods to walk the tree, decide on an
+appropriate response and then build that response using additional helper
+methods on the Node class. In this way, the game server component can work
+entirely on nodes, decoupled from the wire protocol itself and the HTTP request
+details.
+
+For details on how each piece works, see the respective classes as they have
+complete docstrings and type hints.

README.md

@@ -0,0 +1,594 @@
+Introduction
+------------
+A collection of programs for working with various games in the BEMANI series. This
+could be untangled quite a bit into various modules that provide simpler pieces.
+However, this is how it ended up evolving over time. This repository includes
+utilities for unpacking (and sometimes repacking) various file formats, emulating
+network services for various games, utilities for sniffing, redirecting and
+reconstructing network packets, utilities for gathering information about various
+game music databases and associated tooling that makes developing the previous
+utilities easier. It is meant to be a complete ecosystem for somebody looking to
+provide hobby network services to themselves in order to preserve a particular era
+of gaming that is no longer officially supported.
+
+Thanks to Tau for the great writeup on the binary network format. Thanks to some
+rando on stack overflow for RC4 code for Python. Thanks to some other rando on
+stack overflow for sample sniffer code for Python. Thanks to Tau again for the
+great logging in easerver to compare my original output to. Thanks to PKGINGO for
+encouragement and well-received excitement about progress. Thanks to Sarah and Alice
+for being good RE partners for several games, and sharing good finds. Thanks to
+helvetica for helping with game RE and retweeting cute anime ladies onto my feed
+every goddamn night.
+
+2dxutils
+========
+A utility for unpacking and repacking `.2dx` files. This isn't the best utility and
+I think there are more complete and more accurate programs out there. However,
+they all lack source as far as I could tell, so I developed this. Run it like
+`./2dxutils --help` to see help output and determine how to use this.
+
+api
+===
+Development version of this repository's BEMAPI implementation. Run it like
+`./api --help` to see help output and determine how to use this. Much like
+"services" and "frontend", this should be pointed at the development version of
+your services config file, which holds information about the MySQL database that
+this should connect to as well as what game series are supported. See
+`config/server.yaml` for an example file that you can modify.
+
+Do not use this utility to serve production traffic. Instead, see
+`bemani/wsgi/api.wsgi` for a ready-to-go WSGI file that can be used with uWSGI
+and nginx.
+
+arcutils
+========
+A utility for unpacking `.arc` files. This does not currently repack files. However,
+the format is so trivial that adding such a feature would be fairly easy. Run it
+like `./arcutils --help` to see help output and determine how to use this.
+
+bemanishark
+===========
+A wire sniffer that can decode eAmuse packets and print them. Run it on a computer
+that can sniff traffic between an eAmusement server and a supported game and it will
+spit out the requests and responses XML-formatted identically to the legacy easerver
+XML output. This works on both binary and XML traffic. Note that it does not have the
+capability to sniff SSL-encrypted traffic, so don't even bother trying to run this
+at an arcade with official support.
+
+Run it like `sudo ./bemanishark` to invoke. Will run indefinitely until killed
+(Ctrl-C will suffice). Run like `./bemanishark --help for options. Without options,
+it assumes you want to sniff port 80 for all addresses. Note that it doesn't support
+the Base64 binary blob formats found in SN1 and 2. Note also that over time it will
+start to lose packets. This is a bug that I never figured out, and it appears to be
+the OS failing to send over some packets resulting in a failure to reassemble the
+TCP stream.
+
+This utility might be better if rewritten to be a plugin for Wireshark instead of
+a standalone sniffing utility, but I don't have the time.
+
+binutils
+========
+A utility for unpacking raw binxml files (files that use the same encoding scheme
+as the binary network protocol) to their XML representation. This is useful for
+examining raw binary blobs or digging into unknown file formats that contain binxml.
+Run it like `./binutils --help` to see help and learn how to use this.
+
+cardconvert
+===========
+A command-line utility for converting between card numbers written on the back of a
+card and the card ID stored in the RFID of the card. Run it like `./cardconvert --help`
+to see how to use this. This will sanitize input, so you can feed it card numbers
+with or without spaces, and you can mix up 1 and I as well as 0 and O, and it will
+properly handle decoding. This supports both new and old style cards.
+
+dbutils
+=======
+A command-line utility for working with the DB used by "api", "services" and "frontend".
+This utility includes options for creating tables in a newly-created DB, granting and
+revoking admin rights to the frontend, generating migration scripts for live DBs, and
+upgrading live DBs based on previously created migration scripts. Its driven by alembic
+under the hood. You will use `create on initial setup to generate a working MySQL
+database. If you change the schema in code, you can use this again with the `generate`
+option to generate a migration sript. Whenever you run an upgrade to your production
+instance, you should run this against your production DB with the `upgrade` option to
+bring your production DB up to sync with the code you are deploying. Run it like
+`./dbutils --help` to see all options. The config file that this works on is the same
+that is given to "api", "services" and "frontend".
+
+frontend
+========
+Development version of a frontend server allowing for account and server administration
+as well as score viewing and profile editing. Run it like `./frontend --help` to see
+help output and determine how to use this. Much like "services" and "api", this should
+be pointed at the development version of your services config file, which holds
+information about the MySQL database that this should connect to as well as what game
+series are supported. See `config/server.yaml` for an example file that you can modify.
+
+Do not use this utility to serve production traffic. Instead, see
+`bemani/wsgi/frontend.wsgi` for a ready-to-go WSGI file that can be used with uWSGI
+and nginx.
+
+ifsutils
+========
+A mediocre utility that can extract `.ifs` files. This has a lot of baked in
+assumptions and is not nearly as good as other open-source utilities for extracting
+files. It also cannot repack files. This is included for posterity, and because some
+bootstrapping code requires it in order to fully start a production server.
+Run it like `./ifsutils --help` to see help output and learn how to use it.
+
+iidxutils
+=========
+A utility for patching IIDX music database files. Note that this currently can only
+apply a "hide leggendarias from normal folders" patch, although its probable that it
+can be extended for other uses.
+
+proxy
+=====
+A utility to MITM an eAmuse session. Point a game at the port this listens on, and
+point it at another network to see the packets flowing between the two. Takes care
+of rewriting the facility message to MITM all messages. Has the ability to rewrite
+a request/response on the fly which is not currently used except for facility rewriting.
+Its possible that this could be used to on-the-fly patch packets coming back from a
+network which you don't control to do things such as enable paseli and adjust other
+settings that you cannot normally access. Logs in an identical format to bemanishark.
+Useful for black-box RE of other networks. Note that this does not have the ability
+to MITM SSL-encrypted traffic, so don't bother trying to use this on an official network.
+
+This also has the ability to route a packet to one of several known networks based on
+the PCBID, so this can also be used as a proxy for switching networks on the fly.
+With a config file, this can be used as a VIP of sorts, allowing you to point all of
+your games at a single server that runs this proxy, and forward games on a per-PCBID
+basis to various networks behind the scenes. For an example config file to use "proxy"
+as a VIP, see `config/proxy.yaml`. For a more reliable proxy, use the wsgi version
+of this utility located at `bemani/wsgi/proxy.wsgi` along with uWSGI and nginx.
+
+Run it like `./proxy --help` to see how to use this utility.
+
+psmap
+=====
+A utility to take an offset from a DLL file and produce python code that would generate
+a suitable response that said DLL will properly parse. Essentially, if you are
+reversing a new game and they use the `psmap` utility to decode all or part of a
+packet, you can grab either the physical offset into the DLL or the virtual address of
+the data and use this utility to generate the code necessary to service that request.
+Note that this doesn't currently work on 64bit games, but it should be trivial to
+figure out the differences in the 64-bit psmap implementation. Run it like
+`./psmap --help` to see how to use this utility.
+
+read
+====
+A utility to read music DB information out of game files and populate a database.
+This should be given the same config file as "api", "services" or "frontend" and
+assumes that "dbutils" has already been used to instantiate a valid MySQL DB. It
+also assumes you have the correct game files to read out of. Run it like
+`./read --help` to see how to use it. This utility's uses are extensively documented
+below in the "Installation" section.
+
+replay
+======
+A utility to take a packet as logged by proxy, services, trafficgen or bemanishark,
+and replay that packet against a particular server. Useful for quickly grabbing
+packets that caused a crash and debugging the crash (and verifying the fix). It also
+lets you replay that packet against your production instance once you fixed the issue
+in case that packet was a score or profile update that you care about.
+
+responsegen
+===========
+A utility to take a packet as logged by proxy, services, trafficgen or bemanishark,
+and generate python code that would have generated that exact packet. Useful for
+quickly grabbing packets sniffed from another network and prototyping new game support.
+Think of this as a combination of "replay" and "psmap". This is also extremely useful
+when building new integration test clients. Run it like `./responsegen --help` to
+see all information and usage.
+
+scheduler
+=========
+A command-line utility for kicking off scheduled work that must be performed against a
+DB. This includes picking new dailies/weeklies, new courses, etc... depending on the
+game and any requirements that the server perform some actual calculation based on
+time. Essentially, any game backend that includes a `run_scheduled_work` override will
+be acted on by this utility. Note that this takes care of scheduling cadence and
+should be seen as a utility-specific cron handler. You can safely run this repeatedly
+and as frequently as desired. Run like `./scheduler --help` to see how to ues this.
+This should be given the same config file as "api", "frontend" and "services".
+
+services
+========
+Development version of an eAmusement protocol server using flask and the protocol
+libraries also used in "bemanishark" and "trafficgen". Currently it lets most modern
+BEMANI games boot and supports full profile and events for Beatmania IIDX 20-24,
+Pop'n Music 19-24, Jubeat Saucer, Saucer Fulfill, Prop, Qubell and Clan, Sound Voltex
+1, 2, 3 Season 1/2 and 4, Dance Dance Revolution X2, X3, 2013, 2014 and Ace, MÚSECA 1,
+MÚSECA 1+1/2, Reflec Beat, Limelight, Colette, groovin'!! Upper, Volzza 1 and Volzza 2,
+and finally The\*BishiBashi.
+
+Do not use this utility to serve production traffic. Instead, see
+`bemani/wsgi/services.wsgi` for a ready-to-go WSGI file that can be used with uWSGI
+and nginx.
+
+shell
+=====
+A convenience wrapper to invoke a Python 3 shell that has paths set up to import the
+modules in this repository. If you want to tinker or write a quick one-off, this is
+probably the easiest way to do so.
+
+struct
+======
+A convenience utility for helping reverse-engineer structures out of game DLLs. You
+can give this a physical DLL offset or a virtual memory address for the start and
+end of the data as well as a python struct format (documentation at
+https://docs.python.org/3.6/library/struct.html) and this will print the decoded
+data to the screen as a series of tuples. Run it like `./struct --help` to see how
+to use this.
+
+trafficgen
+==========
+A utility for simulating traffic to an eAmusement service. Given a particular game,
+this will run through and attempt to verify simple operation of that service. No
+guarantees are made on the accuracy of the emulation though I've strived to be
+correct. In some cases, I will verify the response, and in other cases I will
+simply verify that certain things exist so as not to crash a real client. This
+currently generates traffic emulating Beatmania IIDX 20-24, Pop'n Music 19-24, Jubeat
+Saucer, Fulfill, Prop, Qubell and Clan, Sound Voltex 1, 2, 3 Season 1/2 and 4, Dance
+Dance Revolution X2, X3, 2013, 2014 and Ace, The\*BishiBashi, MÚSECA 1 and MÚSECA 1+1/2,
+Reflec Beat, Reflec Beat Limelight, Reflec Beat Colette, groovin'!! Upper, Volzza 1 and
+Volzza 2 and can verify card events and score events, as well as PASELI transactions.
+
+verifylibs
+==========
+Unit test frontend utility. This will invoke nosetests on the embarrasingly small
+collection of unit tests for this repository. If you are making modifications, it can
+be useful to write a test first (placed in the `bemani/tests/` directory) and code
+from there. It is also useful when optimizing or profiling, and also to verify that
+you haven't regressed anything.
+
+verifylint
+==========
+Lint invocation utility. This simply invokes flake8 with various options so that you
+can see you haven't introduced any lint errors.
+
+verifytraffic
+=============
+A utility which attempts to call "trafficgen" for each supported game on the network.
+Think of this as a full integration test suite, as it will sweep through each supported
+game and verify that network services are actually working. This assumes that you are
+running "services". Do not point this at a production instance since it **will**
+submit bogus cards, scores, names and the like and mess up your network. This takes
+a config file which sets up how the client should behave. See `config/trafficgen.yaml`
+for a sample file that can be used.
+
+verifytyping
+============
+Typing invocation utility. Since this repository is fully typed, this verifies that you
+haven't introduced any type errors and often catches bugs far faster than attemping to
+play a round only to see that you misused a class or misspelled a variable.
+
+Installation
+------------
+
+Dependency Setup
+================
+The code contained here assumes Python 3.6 as the base. If you don't have or don't
+want to install Python 3.6 as your system python, it is recommended to use
+virtualenv to create a virtual environment. The rest of the installation will assume
+you have Python 3.6 working properly (and are in an activated virtual environment if
+this is the route you've chosen to go). This code is designed to run on Linux.
+However, it has been tested successfully on Windows and OSX as it doesn't use any
+system libraries and sticks to pure python. YMMV in this regard, however, since the
+whole suite is built and tested using a Debian-based derivative.
+
+To install the required libraries, run the following command out of the root of the
+repository. This should allow all of the programs to at least start, but it still
+requires a MySQL database for many of them to be useful. This step has a dependency
+on an isntalled MySQL server and client as well as MySQL client development libraries.
+It also assumes that you've installed the 'wheel' package already. In order to
+compile the mysql client libraries, you will need to have libssl and libcrypto on your
+system as well. To satisfy these requirements on a Debian-based install, run the
+following command:
+
+  sudo apt install libssl-dev zlib1g-dev mysql-server mysql-client libmysqlclient-dev
+
+Once you have all of the above present, run the following command:
+
+  pip install -r requirements.txt
+
+Installing MySQL is outside the scope of this readme, so it is assumed that you have
+a MySQL database with access to create a new DB and tables within it. Note that this
+software requires MySQL version 5.7 or greater. This is due to the extensive use of
+the "json" column type added in 5.7. Create a database (the default database with
+this code is 'bemani') accessed by some user and password (the default user/pass for
+this code is 'bemani'/'bemani'). To create all of the required tables for the
+installation, run the following, substituting the config file for one that you've
+customized if you've done so. The config file that you use here should also be used
+with "api", "services", and "frontend" as well as various other utilities documented
+above.
+
+  ./dbutils --config config/server.yaml create
+
+In order to run the frontend, Python will need to find a javascript runtime. This
+is so it can precompile react components at render time so there doesn't need to be
+a compile step when developing. I found it absolutely bonkers that the backend could
+be on-the-fly reloaded but I had to go through an entire build process to produce
+interpreted JS code, so I went the route of self-contained services instead. Installing
+a JS runtime is also outside the scope of this document, but a quick way to get started
+is to install node.js.
+
+The default configuration points the frontend/backend cache at `/tmp`. It is recommended
+to change to a different directory, as using `/tmp` can cause some items not to be cached.
+This is due to the way `/tmp` on Linux restricts file access to the creator only, so
+if you share your cache with multiple utilities running under different users, it will
+fail to reuse the cache and drastically slow down the frontend.
+
+Database Initialization
+=======================
+At this point, games will boot when pointed at the network, but you won't be able
+to save scores. This is due to the missing song/chart -> score mapping. You will find
+default configuration files for the traffic generator and the services backend in
+the config/ directory. If you've customized your database setup, you will want to
+update the hostname/username/password/database here. You will also want to update
+the server address and frontend URL to customize your instance.
+
+To create the song/chart -> score mapping, you will want to run through the following
+section to import data from each game series. Be sure to substitute your own services
+config in place of the default if you've customized it. Note that if there have been updates
+to the files since you initially imported, you can run with the `--update` flag which
+forces the metadata to be overwritten in the DB instead of skipped. This won't normally
+happen, but if you make improvements to music DB parsing, you will want this to update
+your network.
+
+Note that you'll see a lot of re-used entries. That will happen when the import script
+finds an existing set of charts for the same song in a different game version and links
+the two game versions together. This is how scores can be shared across different versions
+of the same game.
+
+If you happen to already be an authorized client of a BEMAPI-compatible server, you can
+fast-track initializing your server by pointing it at the remote and using its existing
+database to seed your own. If this is the case, run the following command to perform
+a complete initialization. If you wish to update your initial setup with newer data,
+perhaps because a new supported game is available, you can run the following script and
+append the `--update` flag to it. Otherwise, run the following command like so:
+
+  ./bootstrap --config config/server.yaml \
+      --server http://some-server.here/ --token some-token-here
+
+If you do not have a BEMAPI-compatible server, you can initialize the server from the
+game files of the games you wish to run. See the following sections for how exactly to
+do that.
+
+Pop'n Music
+~~~~~~~~~~~
+For Pop'n Music, get the game DLL from the version of the game you want to import and
+run a command like so. This network supports versions 19-24 so you will want to run this
+command once for every version, giving the correct DLL file:
+
+  ./read --config config/server.yaml --series pnm --version 22 --bin popn22.dll
+
+Jubeat
+~~~~~~
+For Jubeat, get the music XML out of the data directory of the mix you are importing,
+and then use "read" with `--series jubeat` and `--version` corresponding to the following
+table:
+
+  Saucer:         saucer
+  Saucer Fulfill: saucer-fulfill
+  Prop:           prop
+  Qubell:         qubell
+  Clan:           clan
+
+An example is as follows:
+
+  ./read --config config/server.yaml --series jubeat --version saucer --xml \
+      music_info.xml
+
+You will also want to populate the Jubeat name database with the following command
+after importing all mixes:
+
+  ./read --config config/server.yaml --series jubeat --version all --tsv \
+      data/jubeat.tsv
+
+IIDX
+~~~~
+For IIDX, you will need the data directory of the mix you wish to support. The import
+script automatically scrapes the music DB as well as the song charts to determine
+difficulty, notecounts and BPM. For a normal mix, you will want to run the command like
+so. This network supports versions 20-24 so you will want to run this command once for
+every version, giving the correct bin file:
+
+  ./read --config config/server.yaml --series iidx --version 22 --bin \
+      gamedata/data/info/music_data.bin --assets gamedata/data/sound/
+
+Note that for omnimix mixes, you will need to point at the omnimix version of
+`music_data.bin`, normally named `music_omni.bin`. For the version, prepend "omni-" to the
+number, like so:
+
+  ./read --config config/server.yaml --series iidx --version omni-22 --bin \
+      gamedata/data/info/music_omni.bin --assets gamedata/data/sound/
+
+You will also want to update the IIDX name database with the following command
+after importing all mixes (this fixes some inconsistencies in names):
+
+  ./read --config config/server.yaml --series iidx --version all --tsv \
+      data/iidx.tsv
+
+DDR
+~~~
+For DDR, you will need the game DLL and `musicdb.xml` from the game you wish to import,
+and then run a command similar to the following. You will want to use the version
+corresponding to version in the following table:
+
+  X2:            12
+  X3 vs. 2ndMix: 13
+  2013:          14
+  2014:          15
+  Ace:           16
+
+  ./read --config config/server.yaml --series ddr --version 15 --bin ddr.dll \
+      --xml data/musicdb.xml
+
+For DDR Ace, there is no `musicdb.xml` or game DLL needed. Instead, you will need the
+`startup.arc` file, like the following example:
+
+  ./read --config config/server.yaml --series ddr --version 16 \
+      --bin data/arc/startup.arc
+
+SDVX
+~~~~
+For SDVX, you will need the game DLL and `music_db.xml` from the game you wish to import,
+and then run the following command, modifying the version parameter as required.
+Note that for SDVX 1, you want the `music_db.xml` file in `data/others/music_db/` directory,
+but for SDVX 2 and onward, you will want the file in `data/others/` instead.
+
+  ./read --config config/server.yaml --series sdvx --version 1 \
+      --xml data/others/music_db.xml
+
+For SDVX 1, you will also need to import the item DB, or appeal cards will not work
+properly. To do so, run the following command.
+
+  ./read --config config/server.yaml --series sdvx --version 1 \
+      --bin soundvoltex.dll
+
+For SDVX 2 and 3, you will also need to import the appeal message DB, or appeal cards
+will not work properly. To do so, run the following command, substituting the correct
+version number.
+
+  ./read --config config/server.yaml --series sdvx --version 2 \
+      --csv data/others/appealmessage.csv
+
+For SDVX 4, you will also need to import the appeal card DB, or appeal cards will not
+work properly. To do so, run the following command.
+
+  ./read --config config/server.yaml --series sdvx --version 4 \
+      --xml data/others/appeal_card.xml
+
+MÚSECA
+~~~~~~
+For MÚSECA, you will need the `music-info.xml` file from the game you wish to import.
+Then, run the following command, modifying the version parameter as required.
+
+  ./read --config config/server.yaml --series museca --version 1 \
+      --xml data/museca/xml/music-info.xml
+
+Reflec Beat
+~~~~~~~~~~~
+For Reflec Beat, get the game DLL from the version of the game you want to import and
+run a command like so. This network supports Reflec Beat up through Volzza 2, so you
+will want to run this with versions 1-6 to completely initialize:
+
+  ./read --config config/server.yaml --series reflec --version 1 --bin reflecbeat.dll
+
+Running Locally
+===============
+Once you've set all of this up, you can start the network in debug mode using a command
+similar to:
+
+  ./services --port 5730 --config config/server.yaml
+
+You can start the frontend in debug mode using another similar command as such:
+
+  ./frontend --port 8573 --config config/server.yaml
+
+You can start the BEMAPI REST server in debug mode using a command similar to:
+
+  ./api --port 18573 --config config/server.yaml
+
+The network config for any particular game should look similar to the following, with
+the correct hostname or IP filled in for the services URL. No path is necessary. Note
+that if you wish to switch between an existing network and one you serve using the
+"proxy" utility, you can set up the services URL to include subdirectories as required
+by that network. This code does not examine nor care about anything after the initial
+slash, so it can be whatever.
+
+  <network>
+    <timeout __type="u32">30000</timeout>
+    <sz_xrpc_buf __type="u32">102400</sz_xrpc_buf>
+    <ssl __type="bool">0</ssl>
+    <services __type="str">http://127.0.0.1:5730/</services>
+  </network>
+
+If you wish to verify the network's operation with some test traffic, feel free to
+point the traffic generator at your development network. You should run it similar to
+the command below, substituting the correct port to connect to your network and choosing
+one of the supported games. If you don't know a supported game, you can use the `--list`
+option to print them. If "Success!" is printed after all checks, you're good to go!
+
+  ./trafficgen --config config/trafficgen.yaml --port 5730 --game pnm-22 && echo Success!
+
+You will want to set up a cron job or similar scheduling agent to call "scheduler"
+on a regular basis. It is recommended to call it every five minutes since there are cache
+warming portions for the front-end that expire every 10 minutes. Game code will register
+with internal handlers to perform daily/weekly actions which are kicked off by this script.
+An example invocation of the tool is as follows:
+
+  ./scheduler --config config/server.yaml
+
+Once your network is up and running, if you pull new code down, the DB schema may have
+changed. For that, use the same DB util script detailed above in the following manner.
+This will walk through all migration scripts that you haven't applied and bring your DB
+up to spec. It is recommended to create a deploy script that knows how to restart uWSGI,
+install a new version of these utilities to your production virtualenv and then runs this
+script to ensure that your production DB is kept in sync:
+
+  ./dbutils --config config/server.yaml upgrade
+
+Since the network provided is player-first, in order to promote an account to administrator
+you will have to create an account on a game first. Once you have done that, you can sign
+up for the front-end using those credentials (your card and PIN), and then use the dbutils
+script to promote yourself to admin, similar to this command:
+
+  ./dbutils --config config/server.yaml add-admin --username <your-name-here>
+
+Production Setup
+================
+As alluded to several times in this README, the recommended way to run a production
+instance of this code is to set up uWSGI fronted by nginx. You should SSL-encrypt
+the frontend and the API services, and its recommended to use LetsEncrypt for a
+free certificate that you can manage easily. There are other ways to run this software
+but I have no experience with or advice on them.
+
+The easiest way to get up and running is to install MySQL 5.7, nginx and uWSGI along
+with Python 3.6 or higher. Create a directory where the services will live and place
+a virtualenv inside it (outside the scope of this document). Then, the wsgi files found
+in `bemani/wsgi/` can be placed in here, uWSGI pointed at them and nginx set up. The
+setup for the top-level package will include all of the frontend templates, so you can
+set up a nginx directory to serve the static resources directly.
+
+For example configurations, an example install script, and an example script to back
+up your MySQL instance, see the `examples/` directory.
+
+Contributing
+------------
+Contributions are welcome! Before submitting a pull request, ensure that your code
+is type-hint clean by running `./verifytyping` and ensure that it hasn't broken basic
+libraries with `./verifylibs`. Make sure that it is also lint-clean with `./verifylint`.
+If you are changing code related to a particular game, it is nice to include a
+verification in the form of a game traffic emulator, so that basic functionality can
+be verified. To ensure you haven't broken another game with your changes, its recommended
+to run the traffic generator against your code with various games. For convenience, you
+can run `./verifytraffic --config config/trafficgen.yaml` to run all supported games
+against your change. Remember that some games require you to run the scheduler to
+generate dailies/weeklies, and if you neglect to run this some of the integration
+tests will fail as they require full packet support! If possible, please also write
+a unit test for your changes. However, if the unit test is just a tautology and an
+integration/traffic test will suit better, then do that instead.
+
+For documentation on how the protocol layer works, see "PROTOCOL". For documentation
+on how the eAmusement server is intended to work, see "BACKEND". Inside `bemani/data/`
+the various DB model files have comments detailing the intended usage of each of the
+tables. For documentation on how the BEMAPI REST API should respond, please see the
+BEMAPI specification repository at https://github.com/DragonMinded/bemapi.
+
+When updating DB schema in the various `bemani/data/` python files, you will most-likely
+want to generate a migration for others to use. For that, we've integrated with alembic
+in order to provide robust migrations. The same DB utility script detailed above will
+create a migration script for you, given a message specifying the operation taking place.
+You should run this **after** making the code change to the schema in the relevant
+file under `bemani/data/mysql`. Alembic will automatically diff your development MySQL
+DB against the schema change you've made and generate an appropriate migration. Sometimes
+you will want to augment that migration with addtional data transformations. Various
+existing migrations do just that, so have a look at them under
+`bemani/data/migrations/versions/`. An example is as follows:
+
+  ./dbutils --config config/server.yaml generate --message "Adding timestamp column to user."
+
+Once the script finishes, check out the created migration script to be sure its correct
+and then check it in.

TODO.md

@@ -0,0 +1,10 @@
+Things that I have not gotten around to doing.
+
+ - IIDX favorites viewer and editor for frontend. The data is all available in profile, but the interface was never built.
+ - DDR calorie tracker and workout stats. Again, the data is all available but the interface was never built.
+ - Rivals for several games. Pop'n and Jubeat come to mind as games that I never supported rivals for, but have support in-game.
+ - Lobby for all games except Reflec Beat. Reflec is the only game with lobby support right now, but this should be fairly easy to add to other games since the backend infra is all there. Correct global IP detection is even built-in to the server and passed forward if you are using a proxy.
+ - Prettify the frontend. Its a bit utilitarian right now, aside from some minor color flare.
+ - Make the frontend work better on mobile. It works well enough, but it could be a much better experience.
+ - Support for DanEvo. I meant to do this but my DanEvo ended up in storage before I could tackle it, so the only thing that exists at the moment is a rudimentary music DB parser.
+ - Figure out phase/unlock/etc bits for some older IIDX and Pop'n Music versions and hook them up to the Arcade panel to allow switching events.

api

@@ -0,0 +1,4 @@
+#! /bin/bash
+
+export PYTHONPATH=$(python -c "import os; print(os.path.realpath('.'))")
+python3 -m bemani.utils.api "$@"

arcutils

@@ -0,0 +1,4 @@
+#! /bin/bash
+
+export PYTHONPATH=$(python -c "import os; print(os.path.realpath('.'))")
+python3 -m bemani.utils.arcutils "$@"

bemani/api/__init__.py

@@ -0,0 +1 @@
+from bemani.api.app import app, config

bemani/api/app.py

@@ -0,0 +1,311 @@
+import copy
+import json
+import traceback
+from typing import Any, Callable, Dict
+from flask import Flask, abort, request, Response, g  # type: ignore
+from functools import wraps
+
+from bemani.api.exceptions import APIException
+from bemani.api.objects import RecordsObject, ProfileObject, StatisticsObject, CatalogObject
+from bemani.common import GameConstants, APIConstants, VersionConstants
+from bemani.data import Data
+
+app = Flask(
+    __name__
+)
+config: Dict[str, Any] = {}
+
+SUPPORTED_VERSIONS = ['v1']
+
+
+def jsonify_response(data: Dict[str, Any], code: int=200) -> Response:
+    return Response(
+        json.dumps(data).encode('utf8'),
+        content_type="application/json; charset=utf-8",
+        status=code,
+    )
+
+
+@app.before_request
+def before_request() -> None:
+    global config
+
+    g.config = config
+    g.data = Data(config)
+    g.authorized = False
+
+    authkey = request.headers.get('Authorization')
+    if authkey is not None:
+        try:
+            authtype, authtoken = authkey.split(' ', 1)
+        except ValueError:
+            authtype = None
+            authtoken = None
+
+        if authtype.lower() == 'token':
+            g.authorized = g.data.local.api.validate_client(authtoken)
+
+
+@app.teardown_request
+def teardown_request(exception: Any) -> None:
+    data = getattr(g, 'data', None)
+    if data is not None:
+        data.close()
+
+
+def authrequired(func: Callable) -> Callable:
+    @wraps(func)
+    def decoratedfunction(*args: Any, **kwargs: Any) -> Response:
+        if not g.authorized:
+            return jsonify_response(
+                {'error': 'Unauthorized client!'},
+                401,
+            )
+        else:
+            return func(*args, **kwargs)
+    return decoratedfunction
+
+
+def jsonify(func: Callable) -> Callable:
+    @wraps(func)
+    def decoratedfunction(*args: Any, **kwargs: Any) -> Response:
+        return jsonify_response(func(*args, **kwargs))
+    return decoratedfunction
+
+
+@app.errorhandler(Exception)
+def server_exception(exception: Any) -> Response:
+    stack = ''.join(traceback.format_exception(type(exception), exception, exception.__traceback__))
+    print(stack)
+    try:
+        g.data.local.network.put_event(
+            'exception',
+            {
+                'service': 'api',
+                'request': request.url,
+                'traceback': stack,
+            },
+        )
+    except Exception:
+        pass
+
+    return jsonify_response(
+        {'error': 'Exception occured while processing request.'},
+        500,
+    )
+
+
+@app.errorhandler(APIException)
+def api_exception(exception: Any) -> Response:
+    return jsonify_response(
+        {'error': exception.message},
+        exception.code,
+    )
+
+
+@app.errorhandler(500)
+def server_error(error: Any) -> Response:
+    return jsonify_response(
+        {'error': 'Exception occured while processing request.'},
+        500,
+    )
+
+
+@app.errorhandler(501)
+def protocol_error(error: Any) -> Response:
+    return jsonify_response(
+        {'error': 'Unsupported protocol version in request.'},
+        501,
+    )
+
+
+@app.errorhandler(400)
+def bad_json(error: Any) -> Response:
+    return jsonify_response(
+        {'error': 'Request JSON could not be decoded.'},
+        500,
+    )
+
+
+@app.errorhandler(404)
+def unrecognized_object(error: Any) -> Response:
+    return jsonify_response(
+        {'error': 'Unrecognized request game/version or object.'},
+        404,
+    )
+
+
+@app.errorhandler(405)
+def invalid_request(error: Any) -> Response:
+    return jsonify_response(
+        {'error': 'Invalid request URI or method.'},
+        405,
+    )
+
+
+@app.route('/<path:path>', methods=['GET', 'POST'])
+@authrequired
+def catch_all(path: str) -> Response:
+    abort(405)
+
+
+@app.route('/', methods=['GET', 'POST'])
+@authrequired
+@jsonify
+def info() -> Dict[str, Any]:
+    requestdata = request.get_json()
+    if requestdata is None:
+        raise APIException('Request JSON could not be decoded.')
+    if requestdata:
+        raise APIException('Unrecognized parameters for request.')
+
+    return {
+        'versions': SUPPORTED_VERSIONS,
+        'name': g.config.get('name', 'e-AMUSEMENT Network'),
+        'email': g.config.get('email', 'nobody@nowhere.com'),
+    }
+
+
+@app.route('/<protoversion>/<requestgame>/<requestversion>', methods=['GET', 'POST'])
+@authrequired
+@jsonify
+def lookup(protoversion: str, requestgame: str, requestversion: str) -> Dict[str, Any]:
+    requestdata = request.get_json()
+    for expected in ['type', 'ids', 'objects']:
+        if expected not in requestdata:
+            raise APIException('Missing parameters for request.')
+    for param in requestdata:
+        if param not in ['type', 'ids', 'objects', 'since', 'until']:
+            raise APIException('Unrecognized parameters for request.')
+
+    args = copy.deepcopy(requestdata)
+    del args['type']
+    del args['ids']
+    del args['objects']
+
+    if protoversion not in SUPPORTED_VERSIONS:
+        # Don't know about this protocol version
+        abort(501)
+
+    # Figure out what games we support based on config, and map those.
+    gamemapping = {}
+    for (gameid, constant) in [
+        ('ddr', GameConstants.DDR),
+        ('iidx', GameConstants.IIDX),
+        ('jubeat', GameConstants.JUBEAT),
+        ('museca', GameConstants.MUSECA),
+        ('popnmusic', GameConstants.POPN_MUSIC),
+        ('reflecbeat', GameConstants.REFLEC_BEAT),
+        ('soundvoltex', GameConstants.SDVX),
+    ]:
+        if g.config.get('support', {}).get(constant, False):
+            gamemapping[gameid] = constant
+    game = gamemapping.get(requestgame)
+    if game is None:
+        # Don't support this game!
+        abort(404)
+
+    if requestversion[0] == 'o':
+        omnimix = True
+        requestversion = requestversion[1:]
+    else:
+        omnimix = False
+
+    version = {
+        GameConstants.DDR: {
+            '12': VersionConstants.DDR_X2,
+            '13': VersionConstants.DDR_X3_VS_2NDMIX,
+            '14': VersionConstants.DDR_2013,
+            '15': VersionConstants.DDR_2014,
+            '16': VersionConstants.DDR_ACE,
+        },
+        GameConstants.IIDX: {
+            '20': VersionConstants.IIDX_TRICORO,
+            '21': VersionConstants.IIDX_SPADA,
+            '22': VersionConstants.IIDX_PENDUAL,
+            '23': VersionConstants.IIDX_COPULA,
+            '24': VersionConstants.IIDX_SINOBUZ,
+            '25': VersionConstants.IIDX_CANNON_BALLERS,
+        },
+        GameConstants.JUBEAT: {
+            '5': VersionConstants.JUBEAT_SAUCER,
+            '5a': VersionConstants.JUBEAT_SAUCER_FULFILL,
+            '6': VersionConstants.JUBEAT_PROP,
+            '7': VersionConstants.JUBEAT_QUBELL,
+            '8': VersionConstants.JUBEAT_CLAN,
+        },
+        GameConstants.MUSECA: {
+            '1': VersionConstants.MUSECA,
+            '1p': VersionConstants.MUSECA_1_PLUS,
+        },
+        GameConstants.POPN_MUSIC: {
+            '19': VersionConstants.POPN_MUSIC_TUNE_STREET,
+            '20': VersionConstants.POPN_MUSIC_FANTASIA,
+            '21': VersionConstants.POPN_MUSIC_SUNNY_PARK,
+            '22': VersionConstants.POPN_MUSIC_LAPISTORIA,
+            '23': VersionConstants.POPN_MUSIC_ECLALE,
+            '24': VersionConstants.POPN_MUSIC_USANEKO,
+        },
+        GameConstants.REFLEC_BEAT: {
+            '1': VersionConstants.REFLEC_BEAT,
+            '2': VersionConstants.REFLEC_BEAT_LIMELIGHT,
+            # We don't support non-final COLETTE, so just return scores for
+            # final colette to any network that asks.
+            '3w': VersionConstants.REFLEC_BEAT_COLETTE,
+            '3sp': VersionConstants.REFLEC_BEAT_COLETTE,
+            '3su': VersionConstants.REFLEC_BEAT_COLETTE,
+            '3a': VersionConstants.REFLEC_BEAT_COLETTE,
+            '3as': VersionConstants.REFLEC_BEAT_COLETTE,
+            # We don't support groovin'!!, so just return upper scores.
+            '4': VersionConstants.REFLEC_BEAT_GROOVIN,
+            '4u': VersionConstants.REFLEC_BEAT_GROOVIN,
+            '5': VersionConstants.REFLEC_BEAT_VOLZZA,
+            '5a': VersionConstants.REFLEC_BEAT_VOLZZA_2,
+            '6': VersionConstants.REFLEC_BEAT_REFLESIA,
+        },
+        GameConstants.SDVX: {
+            '1': VersionConstants.SDVX_BOOTH,
+            '2': VersionConstants.SDVX_INFINITE_INFECTION,
+            '3': VersionConstants.SDVX_GRAVITY_WARS,
+            '4': VersionConstants.SDVX_HEAVENLY_HAVEN,
+        },
+    }.get(game, {}).get(requestversion)
+    if version is None:
+        # Don't support this version!
+        abort(404)
+
+    idtype = requestdata['type']
+    ids = requestdata['ids']
+    if idtype not in [APIConstants.ID_TYPE_CARD, APIConstants.ID_TYPE_SONG, APIConstants.ID_TYPE_INSTANCE, APIConstants.ID_TYPE_SERVER]:
+        raise APIException('Invalid ID type provided!')
+    if idtype == APIConstants.ID_TYPE_CARD and len(ids) == 0:
+        raise APIException('Invalid number of IDs given!')
+    if idtype == APIConstants.ID_TYPE_SONG and len(ids) not in [1, 2]:
+        raise APIException('Invalid number of IDs given!')
+    if idtype == APIConstants.ID_TYPE_INSTANCE and len(ids) != 3:
+        raise APIException('Invalid number of IDs given!')
+    if idtype == APIConstants.ID_TYPE_SERVER and len(ids) != 0:
+        raise APIException('Invalid number of IDs given!')
+
+    responsedata = {}
+    for obj in requestdata['objects']:
+        handler = {
+            'records': RecordsObject,
+            'profile': ProfileObject,
+            'statistics': StatisticsObject,
+            'catalog': CatalogObject,
+        }.get(obj)
+        if handler is None:
+            # Don't support this object type
+            abort(404)
+
+        inst = handler(g.data, game, version, omnimix)
+        try:
+            fetchmethod = getattr(inst, 'fetch_{}'.format(protoversion))
+        except AttributeError:
+            # Don't know how to handle this object for this version
+            abort(501)
+
+        responsedata[obj] = fetchmethod(idtype, ids, args)
+
+    return responsedata

bemani/api/exceptions.py

@@ -0,0 +1,5 @@
+class APIException(Exception):
+
+    def __init__(self, msg: str, code: int=500) -> None:
+        self.message = msg
+        self.code = code

bemani/api/objects/__init__.py

@@ -0,0 +1,5 @@
+from bemani.api.objects.base import BaseObject
+from bemani.api.objects.catalog import CatalogObject
+from bemani.api.objects.records import RecordsObject
+from bemani.api.objects.profile import ProfileObject
+from bemani.api.objects.statistics import StatisticsObject

bemani/api/objects/base.py

@@ -0,0 +1,23 @@
+from typing import List, Any, Dict
+
+from bemani.api.exceptions import APIException
+from bemani.data import Data
+
+
+class BaseObject:
+    """
+    A base class which represents a fetchable API object. Every fetchable object
+    will subclass from this and implement one or more version fetches. These
+    are dynamically looked up by the version number provided by the client, so
+    objects can control which versions they reply to by subclassing or ignoring
+    various fetch versions.
+    """
+
+    def __init__(self, data: Data, game: str, version: int, omnimix: bool) -> None:
+        self.data = data
+        self.game = game
+        self.version = version
+        self.omnimix = omnimix
+
+    def fetch_v1(self, idtype: str, ids: List[str], params: Dict[str, Any]) -> Any:
+        raise APIException('Object fetch not supported for this version!')

bemani/api/objects/catalog.py

@@ -0,0 +1,185 @@
+from typing import Any, Dict, List
+
+from bemani.api.exceptions import APIException
+from bemani.api.objects.base import BaseObject
+from bemani.common import GameConstants, APIConstants, DBConstants, VersionConstants
+from bemani.data import Song
+
+
+class CatalogObject(BaseObject):
+
+    def __format_ddr_song(self, song: Song) -> Dict[str, Any]:
+        groove = song.data.get_dict('groove')
+        return {
+            'editid': str(song.data.get_int('edit_id')),
+            'difficulty': song.data.get_int('difficulty'),
+            'bpm_min': song.data.get_int('bpm_min'),
+            'bpm_max': song.data.get_int('bpm_max'),
+            'category': str(song.data.get_int('category')),
+            'groove': {
+                'air': groove.get_int('air'),
+                'chaos': groove.get_int('chaos'),
+                'freeze': groove.get_int('freeze'),
+                'stream': groove.get_int('stream'),
+                'voltage': groove.get_int('voltage'),
+            },
+        }
+
+    def __format_iidx_song(self, song: Song) -> Dict[str, Any]:
+        return {
+            'difficulty': song.data.get_int('difficulty'),
+            'bpm_min': song.data.get_int('bpm_min'),
+            'bpm_max': song.data.get_int('bpm_max'),
+            'notecount': song.data.get_int('notecount'),
+            'category': str(int(song.id / 1000)),
+        }
+
+    def __format_jubeat_song(self, song: Song) -> Dict[str, Any]:
+        return {
+            'difficulty': song.data.get_int('difficulty'),
+            'bpm_min': song.data.get_int('bpm_min'),
+            'bpm_max': song.data.get_int('bpm_max'),
+        }
+
+    def __format_museca_song(self, song: Song) -> Dict[str, Any]:
+        return {
+            'difficulty': song.data.get_int('difficulty'),
+            'bpm_min': song.data.get_int('bpm_min'),
+            'bpm_max': song.data.get_int('bpm_max'),
+            'limited': song.data.get_int('limited'),
+        }
+
+    def __format_popn_song(self, song: Song) -> Dict[str, Any]:
+        return {
+            'difficulty': song.data.get_int('difficulty'),
+            'category': song.data.get_str('category'),
+        }
+
+    def __format_reflec_song(self, song: Song) -> Dict[str, Any]:
+        return {
+            'difficulty': song.data.get_int('difficulty'),
+            'category': str(song.data.get_int('folder')),
+            'musicid': song.data.get_str('chart_id'),
+        }
+
+    def __format_sdvx_song(self, song: Song) -> Dict[str, Any]:
+        return {
+            'difficulty': song.data.get_int('difficulty'),
+            'bpm_min': song.data.get_int('bpm_min'),
+            'bpm_max': song.data.get_int('bpm_max'),
+            'limited': song.data.get_int('limited'),
+        }
+
+    def __format_song(self, song: Song) -> Dict[str, Any]:
+        base = {
+            'song': str(song.id),
+            'chart': str(song.chart),
+            'title': song.name or "",
+            'artist': song.artist or "",
+            'genre': song.genre or "",
+        }
+
+        if self.game == GameConstants.DDR:
+            base.update(self.__format_ddr_song(song))
+        if self.game == GameConstants.IIDX:
+            base.update(self.__format_iidx_song(song))
+        if self.game == GameConstants.JUBEAT:
+            base.update(self.__format_jubeat_song(song))
+        if self.game == GameConstants.MUSECA:
+            base.update(self.__format_museca_song(song))
+        if self.game == GameConstants.POPN_MUSIC:
+            base.update(self.__format_popn_song(song))
+        if self.game == GameConstants.REFLEC_BEAT:
+            base.update(self.__format_reflec_song(song))
+        if self.game == GameConstants.SDVX:
+            base.update(self.__format_sdvx_song(song))
+
+        return base
+
+    def __format_sdvx_extras(self) -> Dict[str, List[Dict[str, Any]]]:
+        # Gotta look up the unlock catalog
+        items = self.data.local.game.get_items(self.game, self.version)
+
+        # Format it depending on the version
+        if self.version == 1:
+            return {
+                "purchases": [
+                    {
+                        "catalogid": str(item.id),
+                        "song": str(item.data.get_int("musicid")),
+                        "chart": str(item.data.get_int("chart")),
+                        "price": item.data.get_int("blocks"),
+                    }
+                    for item in items
+                    if item.type == "song_unlock"
+                ],
+                "appealcards": [],
+            }
+        else:
+            return {
+                "purchases": [],
+                "appealcards": [
+                    {
+                        "appealid": str(item.id),
+                        "description": item.data.get_str("description"),
+                    }
+                    for item in items
+                    if item.type == "appealcard"
+                ],
+            }
+
+    def __format_extras(self) -> Dict[str, List[Dict[str, Any]]]:
+        if self.game == GameConstants.SDVX:
+            return self.__format_sdvx_extras()
+        else:
+            return {}
+
+    @property
+    def music_version(self) -> int:
+        if self.game == GameConstants.IIDX:
+            if self.omnimix:
+                return self.version + DBConstants.OMNIMIX_VERSION_BUMP
+            else:
+                return self.version
+        else:
+            return self.version
+
+    def fetch_v1(self, idtype: str, ids: List[str], params: Dict[str, Any]) -> Dict[str, List[Dict[str, Any]]]:
+        # Verify IDs
+        if idtype != APIConstants.ID_TYPE_SERVER:
+            raise APIException(
+                'Unsupported ID for lookup!',
+                405,
+            )
+
+        # Fetch the songs
+        songs = self.data.local.music.get_all_songs(self.game, self.music_version)
+        if self.game == GameConstants.JUBEAT and self.version == VersionConstants.JUBEAT_CLAN:
+            # There's always a special case. We don't store all music IDs since those in
+            # the range of 80000301-80000347 are actually the same song, but copy-pasted
+            # for different prefectures and slightly different charts. So, we need to copy
+            # that song data so that remote clients can resolve scores for those ID ranges.
+            additions: List[Song] = []
+            for song in songs:
+                if song.id == 80000301:
+                    for idrange in range(80000302, 80000348):
+                        additions.append(
+                            Song(
+                                song.game,
+                                song.version,
+                                idrange,
+                                song.chart,
+                                song.name,
+                                song.artist,
+                                song.genre,
+                                song.data,
+                            )
+                        )
+            songs.extend(additions)
+        retval = {
+            'songs': [self.__format_song(song) for song in songs],
+        }
+
+        # Fetch any optional extras per-game, return
+        retval.update(self.__format_extras())
+        return retval

bemani/api/objects/profile.py

@@ -0,0 +1,131 @@
+from typing import Any, Dict, List, Set, Tuple
+
+from bemani.api.exceptions import APIException
+from bemani.api.objects.base import BaseObject
+from bemani.common import ValidatedDict, GameConstants, APIConstants
+from bemani.data import UserID
+
+
+class ProfileObject(BaseObject):
+
+    def __format_ddr_profile(self, profile: ValidatedDict, exact: bool) -> Dict[str, Any]:
+        return {
+            'area': profile.get_int('area', -1) if exact else -1,
+        }
+
+    def __format_iidx_profile(self, profile: ValidatedDict, exact: bool) -> Dict[str, Any]:
+        qpro = profile.get_dict('qpro')
+
+        return {
+            'area': profile.get_int('pid', -1),
+            'qpro': {
+                'head': qpro.get_int('head', -1) if exact else -1,
+                'hair': qpro.get_int('hair', -1) if exact else -1,
+                'face': qpro.get_int('face', -1) if exact else -1,
+                'body': qpro.get_int('body', -1) if exact else -1,
+                'hand': qpro.get_int('hand', -1) if exact else -1,
+            }
+        }
+
+    def __format_jubeat_profile(self, profile: ValidatedDict, exact: bool) -> Dict[str, Any]:
+        return {}
+
+    def __format_museca_profile(self, profile: ValidatedDict, exact: bool) -> Dict[str, Any]:
+        return {}
+
+    def __format_popn_profile(self, profile: ValidatedDict, exact: bool) -> Dict[str, Any]:
+        return {
+            'character': profile.get_int('chara', -1) if exact else -1,
+        }
+
+    def __format_reflec_profile(self, profile: ValidatedDict, exact: bool) -> Dict[str, Any]:
+        return {
+            'icon': profile.get_dict('config').get_int('icon_id', -1) if exact else -1,
+        }
+
+    def __format_sdvx_profile(self, profile: ValidatedDict, exact: bool) -> Dict[str, Any]:
+        return {}
+
+    def __format_profile(self, cardids: List[str], profile: ValidatedDict, settings: ValidatedDict, exact: bool) -> Dict[str, Any]:
+        base = {
+            'name': profile.get_str('name'),
+            'cards': cardids,
+            'registered': settings.get_int('first_play_timestamp', -1),
+            'updated': settings.get_int('last_play_timestamp', -1),
+            'plays': settings.get_int('total_plays', -1),
+            'match': 'exact' if exact else 'partial',
+        }
+
+        if self.game == GameConstants.DDR:
+            base.update(self.__format_ddr_profile(profile, exact))
+        if self.game == GameConstants.IIDX:
+            base.update(self.__format_iidx_profile(profile, exact))
+        if self.game == GameConstants.JUBEAT:
+            base.update(self.__format_jubeat_profile(profile, exact))
+        if self.game == GameConstants.MUSECA:
+            base.update(self.__format_museca_profile(profile, exact))
+        if self.game == GameConstants.POPN_MUSIC:
+            base.update(self.__format_popn_profile(profile, exact))
+        if self.game == GameConstants.REFLEC_BEAT:
+            base.update(self.__format_reflec_profile(profile, exact))
+        if self.game == GameConstants.SDVX:
+            base.update(self.__format_sdvx_profile(profile, exact))
+
+        return base
+
+    def fetch_v1(self, idtype: str, ids: List[str], params: Dict[str, Any]) -> List[Dict[str, Any]]:
+        # Fetch the profiles
+        profiles: List[Tuple[UserID, ValidatedDict]] = []
+        if idtype == APIConstants.ID_TYPE_SERVER:
+            profiles.extend(self.data.local.user.get_all_profiles(self.game, self.version))
+        elif idtype == APIConstants.ID_TYPE_SONG:
+            raise APIException(
+                'Unsupported ID for lookup!',
+                405,
+            )
+        elif idtype == APIConstants.ID_TYPE_INSTANCE:
+            raise APIException(
+                'Unsupported ID for lookup!',
+                405,
+            )
+        elif idtype == APIConstants.ID_TYPE_CARD:
+            users: Set[UserID] = set()
+            for cardid in ids:
+                userid = self.data.local.user.from_cardid(cardid)
+                if userid is not None:
+                    # Don't duplicate loads for users with multiple card IDs if multiples
+                    # of those IDs are requested.
+                    if userid in users:
+                        continue
+                    users.add(userid)
+
+                    # We can possibly find another profile for this user. This is important
+                    # in the case that we returned scores for a user that doesn't have a
+                    # profile on a particular version. We allow that on this network, so in
+                    # order to not break remote networks, try our best to return any profile.
+                    profile = self.data.local.user.get_any_profile(self.game, self.version, userid)
+                    if profile is not None:
+                        profiles.append((userid, profile))
+        else:
+            raise APIException('Invalid ID type!')
+
+        # Now, fetch the users, and filter out profiles belonging to orphaned users
+        retval: List[Dict[str, Any]] = []
+        id_to_cards: Dict[UserID, List[str]] = {}
+        for (userid, profile) in profiles:
+            if userid not in id_to_cards:
+                cards = self.data.local.user.get_cards(userid)
+                if len(cards) == 0:
+                    # Can't add this user, skip the profile
+                    continue
+
+                id_to_cards[userid] = cards
+
+            # Format the profile and add it
+            settings = self.data.local.game.get_settings(self.game, userid)
+            if settings is None:
+                settings = ValidatedDict({})
+
+            retval.append(self.__format_profile(id_to_cards[userid], profile, settings, profile['version'] == self.version))
+
+        return retval

bemani/api/objects/records.py

@@ -0,0 +1,299 @@
+from typing import Any, Dict, List, Set, Tuple
+
+from bemani.api.exceptions import APIException
+from bemani.api.objects.base import BaseObject
+from bemani.common import GameConstants, VersionConstants, APIConstants, DBConstants
+from bemani.data import Score, UserID
+
+
+class RecordsObject(BaseObject):
+
+    def __format_ddr_record(self, record: Score) -> Dict[str, Any]:
+        halo = {
+            DBConstants.DDR_HALO_NONE: 'none',
+            DBConstants.DDR_HALO_GOOD_FULL_COMBO: 'gfc',
+            DBConstants.DDR_HALO_GREAT_FULL_COMBO: 'fc',
+            DBConstants.DDR_HALO_PERFECT_FULL_COMBO: 'pfc',
+            DBConstants.DDR_HALO_MARVELOUS_FULL_COMBO: 'mfc',
+        }.get(record.data.get_int('halo'), 'none')
+        rank = {
+            DBConstants.DDR_RANK_AAA: "AAA",
+            DBConstants.DDR_RANK_AA_PLUS: "AA+",
+            DBConstants.DDR_RANK_AA: "AA",
+            DBConstants.DDR_RANK_AA_MINUS: "AA-",
+            DBConstants.DDR_RANK_A_PLUS: "A+",
+            DBConstants.DDR_RANK_A: "A",
+            DBConstants.DDR_RANK_A_MINUS: "A-",
+            DBConstants.DDR_RANK_B_PLUS: "B+",
+            DBConstants.DDR_RANK_B: "B",
+            DBConstants.DDR_RANK_B_MINUS: "B-",
+            DBConstants.DDR_RANK_C_PLUS: "C+",
+            DBConstants.DDR_RANK_C: "C",
+            DBConstants.DDR_RANK_C_MINUS: "C-",
+            DBConstants.DDR_RANK_D_PLUS: "D+",
+            DBConstants.DDR_RANK_D: "D",
+            DBConstants.DDR_RANK_E: "E",
+        }.get(record.data.get_int('rank'), 'E')
+
+        if self.version == VersionConstants.DDR_ACE:
+            # DDR Ace is specia
+            ghost = [int(x) for x in record.data.get_str('ghost')]
+        else:
+            if 'trace' not in record.data:
+                ghost = []
+            else:
+                ghost = record.data.get_int_array('trace', len(record.data['trace']))
+
+        return {
+            'rank': rank,
+            'halo': halo,
+            'combo': record.data.get_int('combo'),
+            'ghost': ghost,
+        }
+
+    def __format_iidx_record(self, record: Score) -> Dict[str, Any]:
+        status = {
+            DBConstants.IIDX_CLEAR_STATUS_NO_PLAY: 'np',
+            DBConstants.IIDX_CLEAR_STATUS_FAILED: 'failed',
+            DBConstants.IIDX_CLEAR_STATUS_ASSIST_CLEAR: 'ac',
+            DBConstants.IIDX_CLEAR_STATUS_EASY_CLEAR: 'ec',
+            DBConstants.IIDX_CLEAR_STATUS_CLEAR: 'nc',
+            DBConstants.IIDX_CLEAR_STATUS_HARD_CLEAR: 'hc',
+            DBConstants.IIDX_CLEAR_STATUS_EX_HARD_CLEAR: 'exhc',
+            DBConstants.IIDX_CLEAR_STATUS_FULL_COMBO: 'fc',
+        }.get(record.data.get_int('clear_status'), 'np')
+
+        return {
+            'status': status,
+            'miss': record.data.get_int('miss_count', -1),
+            'ghost': [b for b in record.data.get_bytes('ghost')],
+            'pgreat': record.data.get_int('pgreats', -1),
+            'great': record.data.get_int('greats', -1),
+        }
+
+    def __format_jubeat_record(self, record: Score) -> Dict[str, Any]:
+        status = {
+            DBConstants.JUBEAT_PLAY_MEDAL_FAILED: 'failed',
+            DBConstants.JUBEAT_PLAY_MEDAL_CLEARED: 'cleared',
+            DBConstants.JUBEAT_PLAY_MEDAL_NEARLY_FULL_COMBO: 'nfc',
+            DBConstants.JUBEAT_PLAY_MEDAL_FULL_COMBO: 'fc',
+            DBConstants.JUBEAT_PLAY_MEDAL_NEARLY_EXCELLENT: 'nec',
+            DBConstants.JUBEAT_PLAY_MEDAL_EXCELLENT: 'exc',
+        }.get(record.data.get_int('medal'), 'failed')
+        if 'ghost' not in record.data:
+            ghost: List[int] = []
+        else:
+            ghost = record.data.get_int_array('ghost', len(record.data['ghost']))
+
+        return {
+            'status': status,
+            'combo': record.data.get_int('combo', -1),
+            'ghost': ghost,
+        }
+
+    def __format_museca_record(self, record: Score) -> Dict[str, Any]:
+        rank = {
+            DBConstants.MUSECA_GRADE_DEATH: 'death',
+            DBConstants.MUSECA_GRADE_POOR: 'poor',
+            DBConstants.MUSECA_GRADE_MEDIOCRE: 'mediocre',
+            DBConstants.MUSECA_GRADE_GOOD: 'good',
+            DBConstants.MUSECA_GRADE_GREAT: 'great',
+            DBConstants.MUSECA_GRADE_EXCELLENT: 'excellent',
+            DBConstants.MUSECA_GRADE_SUPERB: 'superb',
+            DBConstants.MUSECA_GRADE_MASTERPIECE: 'masterpiece',
+            DBConstants.MUSECA_GRADE_PERFECT: 'perfect'
+        }.get(record.data.get_int('grade'), 'death')
+        status = {
+            DBConstants.MUSECA_CLEAR_TYPE_FAILED: 'failed',
+            DBConstants.MUSECA_CLEAR_TYPE_CLEARED: 'cleared',
+            DBConstants.MUSECA_CLEAR_TYPE_FULL_COMBO: 'fc',
+        }.get(record.data.get_int('clear_type'), 'failed')
+
+        return {
+            'rank': rank,
+            'status': status,
+            'combo': record.data.get_int('combo', -1),
+            'buttonrate': record.data.get_dict('stats').get_int('btn_rate'),
+            'longrate': record.data.get_dict('stats').get_int('long_rate'),
+            'volrate': record.data.get_dict('stats').get_int('vol_rate'),
+        }
+
+    def __format_popn_record(self, record: Score) -> Dict[str, Any]:
+        status = {
+            DBConstants.POPN_MUSIC_PLAY_MEDAL_CIRCLE_FAILED: 'cf',
+            DBConstants.POPN_MUSIC_PLAY_MEDAL_DIAMOND_FAILED: 'df',
+            DBConstants.POPN_MUSIC_PLAY_MEDAL_STAR_FAILED: 'sf',
+            DBConstants.POPN_MUSIC_PLAY_MEDAL_EASY_CLEAR: 'ec',
+            DBConstants.POPN_MUSIC_PLAY_MEDAL_CIRCLE_CLEARED: 'cc',
+            DBConstants.POPN_MUSIC_PLAY_MEDAL_DIAMOND_CLEARED: 'dc',
+            DBConstants.POPN_MUSIC_PLAY_MEDAL_STAR_CLEARED: 'sc',
+            DBConstants.POPN_MUSIC_PLAY_MEDAL_CIRCLE_FULL_COMBO: 'cfc',
+            DBConstants.POPN_MUSIC_PLAY_MEDAL_DIAMOND_FULL_COMBO: 'dfc',
+            DBConstants.POPN_MUSIC_PLAY_MEDAL_STAR_FULL_COMBO: 'sfc',
+            DBConstants.POPN_MUSIC_PLAY_MEDAL_PERFECT: 'p',
+        }.get(record.data.get_int('medal'), 'cf')
+
+        return {
+            'status': status,
+            'combo': record.data.get_int('combo', -1),
+        }
+
+    def __format_reflec_record(self, record: Score) -> Dict[str, Any]:
+        status = {
+            DBConstants.REFLEC_BEAT_CLEAR_TYPE_NO_PLAY: 'np',
+            DBConstants.REFLEC_BEAT_CLEAR_TYPE_FAILED: 'failed',
+            DBConstants.REFLEC_BEAT_CLEAR_TYPE_CLEARED: 'cleared',
+            DBConstants.REFLEC_BEAT_CLEAR_TYPE_HARD_CLEARED: 'hc',
+            DBConstants.REFLEC_BEAT_CLEAR_TYPE_S_HARD_CLEARED: 'shc',
+        }.get(record.data.get_int('clear_type'), 'np')
+        halo = {
+            DBConstants.REFLEC_BEAT_COMBO_TYPE_NONE: 'none',
+            DBConstants.REFLEC_BEAT_COMBO_TYPE_ALMOST_COMBO: 'ac',
+            DBConstants.REFLEC_BEAT_COMBO_TYPE_FULL_COMBO: 'fc',
+            DBConstants.REFLEC_BEAT_COMBO_TYPE_FULL_COMBO_ALL_JUST: 'fcaj',
+        }.get(record.data.get_int('combo_type'), 'none')
+
+        return {
+            'rate': record.data.get_int('achievement_rate'),
+            'status': status,
+            'halo': halo,
+            'combo': record.data.get_int('combo', -1),
+            'miss': record.data.get_int('miss_count', -1),
+        }
+
+    def __format_sdvx_record(self, record: Score) -> Dict[str, Any]:
+        status = {
+            DBConstants.SDVX_CLEAR_TYPE_NO_PLAY: 'np',
+            DBConstants.SDVX_CLEAR_TYPE_FAILED: 'failed',
+            DBConstants.SDVX_CLEAR_TYPE_CLEAR: 'cleared',
+            DBConstants.SDVX_CLEAR_TYPE_HARD_CLEAR: 'hc',
+            DBConstants.SDVX_CLEAR_TYPE_ULTIMATE_CHAIN: 'uc',
+            DBConstants.SDVX_CLEAR_TYPE_PERFECT_ULTIMATE_CHAIN: 'puc',
+        }.get(record.data.get_int('clear_type'), 'np')
+        rank = {
+            DBConstants.SDVX_GRADE_NO_PLAY: 'E',
+            DBConstants.SDVX_GRADE_D: 'D',
+            DBConstants.SDVX_GRADE_C: 'C',
+            DBConstants.SDVX_GRADE_B: 'B',
+            DBConstants.SDVX_GRADE_A: 'A',
+            DBConstants.SDVX_GRADE_A_PLUS: 'A+',
+            DBConstants.SDVX_GRADE_AA: 'AA',
+            DBConstants.SDVX_GRADE_AA_PLUS: 'AA+',
+            DBConstants.SDVX_GRADE_AAA: 'AAA',
+            DBConstants.SDVX_GRADE_AAA_PLUS: 'AAA+',
+            DBConstants.SDVX_GRADE_S: 'S',
+        }.get(record.data.get_int('grade'), 'E')
+
+        return {
+            'status': status,
+            'rank': rank,
+            'combo': record.data.get_int('combo', -1),
+            'buttonrate': record.data.get_dict('stats').get_int('btn_rate'),
+            'longrate': record.data.get_dict('stats').get_int('long_rate'),
+            'volrate': record.data.get_dict('stats').get_int('vol_rate'),
+        }
+
+    def __format_record(self, cardids: List[str], record: Score) -> Dict[str, Any]:
+        base = {
+            'cards': cardids,
+            'song': str(record.id),
+            'chart': str(record.chart),
+            'points': record.points,
+            'timestamp': record.timestamp,
+            'updated': record.update,
+        }
+
+        if self.game == GameConstants.DDR:
+            base.update(self.__format_ddr_record(record))
+        if self.game == GameConstants.IIDX:
+            base.update(self.__format_iidx_record(record))
+        if self.game == GameConstants.JUBEAT:
+            base.update(self.__format_jubeat_record(record))
+        if self.game == GameConstants.MUSECA:
+            base.update(self.__format_museca_record(record))
+        if self.game == GameConstants.POPN_MUSIC:
+            base.update(self.__format_popn_record(record))
+        if self.game == GameConstants.REFLEC_BEAT:
+            base.update(self.__format_reflec_record(record))
+        if self.game == GameConstants.SDVX:
+            base.update(self.__format_sdvx_record(record))
+
+        return base
+
+    @property
+    def music_version(self) -> int:
+        if self.game == GameConstants.IIDX:
+            if self.omnimix:
+                return self.version + DBConstants.OMNIMIX_VERSION_BUMP
+            else:
+                return self.version
+        else:
+            return self.version
+
+    def fetch_v1(self, idtype: str, ids: List[str], params: Dict[str, Any]) -> List[Dict[str, Any]]:
+        since = params.get('since')
+        until = params.get('until')
+
+        # Fetch the scores
+        records: List[Tuple[UserID, Score]] = []
+        if idtype == APIConstants.ID_TYPE_SERVER:
+            # Because of the way this query works, we can't apply since/until to it directly.
+            # If we did, it would miss higher scores earned before since or after until, and
+            # incorrectly report records.
+            records.extend(self.data.local.music.get_all_records(self.game, self.music_version))
+        elif idtype == APIConstants.ID_TYPE_SONG:
+            if len(ids) == 1:
+                songid = int(ids[0])
+                chart = None
+            else:
+                songid = int(ids[0])
+                chart = int(ids[1])
+            records.extend(self.data.local.music.get_all_scores(self.game, self.music_version, songid=songid, songchart=chart, since=since, until=until))
+        elif idtype == APIConstants.ID_TYPE_INSTANCE:
+            songid = int(ids[0])
+            chart = int(ids[1])
+            cardid = ids[2]
+            userid = self.data.local.user.from_cardid(cardid)
+            if userid is not None:
+                score = self.data.local.music.get_score(self.game, self.music_version, userid, songid, chart)
+                if score is not None:
+                    records.append((userid, score))
+        elif idtype == APIConstants.ID_TYPE_CARD:
+            users: Set[UserID] = set()
+            for cardid in ids:
+                userid = self.data.local.user.from_cardid(cardid)
+                if userid is not None:
+                    # Don't duplicate loads for users with multiple card IDs if multiples
+                    # of those IDs are requested.
+                    if userid in users:
+                        continue
+                    users.add(userid)
+
+                    records.extend([(userid, score) for score in self.data.local.music.get_scores(self.game, self.music_version, userid, since=since, until=until)])
+        else:
+            raise APIException('Invalid ID type!')
+
+        # Now, fetch the users, and filter out scores belonging to orphaned users
+        id_to_cards: Dict[UserID, List[str]] = {}
+        retval: List[Dict[str, Any]] = []
+        for (userid, record) in records:
+            # Postfilter for queries that can't filter. This will save on data transferred.
+            if since is not None:
+                if record.update < since:
+                    continue
+            if until is not None:
+                if record.update >= until:
+                    continue
+
+            if userid not in id_to_cards:
+                cards = self.data.local.user.get_cards(userid)
+                if len(cards) == 0:
+                    # Can't add this user, skip the score
+                    continue
+
+                id_to_cards[userid] = cards
+
+            # Format the score and add it
+            retval.append(self.__format_record(id_to_cards[userid], record))
+
+        return retval

bemani/api/objects/statistics.py

@@ -0,0 +1,230 @@
+from typing import List, Dict, Tuple, Any
+
+from bemani.api.exceptions import APIException
+from bemani.api.objects.base import BaseObject
+from bemani.common import APIConstants, DBConstants, GameConstants
+from bemani.data import Attempt, UserID
+
+
+class StatisticsObject(BaseObject):
+
+    def __format_statistics(self, stats: Dict[str, Any]) -> Dict[str, Any]:
+        return {
+            'cards': [],
+            'song': str(stats['id']),
+            'chart': str(stats['chart']),
+            'plays': stats.get('plays', -1),
+            'clears': stats.get('clears', -1),
+            'combos': stats.get('combos', -1),
+        }
+
+    def __format_user_statistics(self, cardids: List[str], stats: Dict[str, Any]) -> Dict[str, Any]:
+        base = self.__format_statistics(stats)
+        base['cards'] = cardids
+        return base
+
+    @property
+    def music_version(self) -> int:
+        if self.game == GameConstants.IIDX:
+            if self.omnimix:
+                return self.version + DBConstants.OMNIMIX_VERSION_BUMP
+            else:
+                return self.version
+        else:
+            return self.version
+
+    def __is_play(self, attempt: Attempt) -> bool:
+        if self.game in [
+            GameConstants.DDR,
+            GameConstants.JUBEAT,
+            GameConstants.MUSECA,
+            GameConstants.POPN_MUSIC,
+        ]:
+            return True
+        if self.game == GameConstants.IIDX:
+            return attempt.data.get_int('clear_status') != DBConstants.IIDX_CLEAR_STATUS_NO_PLAY
+        if self.game == GameConstants.REFLEC_BEAT:
+            return attempt.data.get_int('clear_type') != DBConstants.REFLEC_BEAT_CLEAR_TYPE_NO_PLAY
+        if self.game == GameConstants.SDVX:
+            return attempt.data.get_int('clear_type') != DBConstants.SDVX_CLEAR_TYPE_NO_PLAY
+
+        return False
+
+    def __is_clear(self, attempt: Attempt) -> bool:
+        if not self.__is_play(attempt):
+            return False
+
+        if self.game == GameConstants.DDR:
+            return attempt.data.get_int('rank') != DBConstants.DDR_RANK_E
+        if self.game == GameConstants.IIDX:
+            return attempt.data.get_int('clear_status') != DBConstants.IIDX_CLEAR_STATUS_FAILED
+        if self.game == GameConstants.JUBEAT:
+            return attempt.data.get_int('medal') != DBConstants.JUBEAT_PLAY_MEDAL_FAILED
+        if self.game == GameConstants.MUSECA:
+            return attempt.data.get_int('clear_type') != DBConstants.MUSECA_CLEAR_TYPE_FAILED
+        if self.game == GameConstants.POPN_MUSIC:
+            return attempt.data.get_int('medal') not in [
+                DBConstants.POPN_MUSIC_PLAY_MEDAL_CIRCLE_FAILED,
+                DBConstants.POPN_MUSIC_PLAY_MEDAL_DIAMOND_FAILED,
+                DBConstants.POPN_MUSIC_PLAY_MEDAL_STAR_FAILED,
+            ]
+        if self.game == GameConstants.REFLEC_BEAT:
+            return attempt.data.get_int('clear_type') != DBConstants.REFLEC_BEAT_CLEAR_TYPE_FAILED
+        if self.game == GameConstants.SDVX:
+            return (
+                attempt.data.get_int('grade') != DBConstants.SDVX_GRADE_NO_PLAY and
+                attempt.data.get_int('clear_type') not in [
+                    DBConstants.SDVX_CLEAR_TYPE_NO_PLAY,
+                    DBConstants.SDVX_CLEAR_TYPE_FAILED,
+                ]
+            )
+
+        return False
+
+    def __is_combo(self, attempt: Attempt) -> bool:
+        if not self.__is_play(attempt):
+            return False
+
+        if self.game == GameConstants.DDR:
+            return attempt.data.get_int('halo') != DBConstants.DDR_HALO_NONE
+        if self.game == GameConstants.IIDX:
+            return attempt.data.get_int('clear_status') == DBConstants.IIDX_CLEAR_STATUS_FULL_COMBO
+        if self.game == GameConstants.JUBEAT:
+            return attempt.data.get_int('medal') in [
+                DBConstants.JUBEAT_PLAY_MEDAL_FULL_COMBO,
+                DBConstants.JUBEAT_PLAY_MEDAL_NEARLY_EXCELLENT,
+                DBConstants.JUBEAT_PLAY_MEDAL_EXCELLENT,
+            ]
+        if self.game == GameConstants.MUSECA:
+            return attempt.data.get_int('clear_type') == DBConstants.MUSECA_CLEAR_TYPE_FULL_COMBO
+        if self.game == GameConstants.POPN_MUSIC:
+            return attempt.data.get_int('medal') in [
+                DBConstants.POPN_MUSIC_PLAY_MEDAL_CIRCLE_FULL_COMBO,
+                DBConstants.POPN_MUSIC_PLAY_MEDAL_DIAMOND_FULL_COMBO,
+                DBConstants.POPN_MUSIC_PLAY_MEDAL_STAR_FULL_COMBO,
+                DBConstants.POPN_MUSIC_PLAY_MEDAL_PERFECT,
+            ]
+        if self.game == GameConstants.REFLEC_BEAT:
+            return attempt.data.get_int('combo_type') in [
+                DBConstants.REFLEC_BEAT_COMBO_TYPE_FULL_COMBO,
+                DBConstants.REFLEC_BEAT_COMBO_TYPE_FULL_COMBO_ALL_JUST,
+            ]
+        if self.game == GameConstants.SDVX:
+            return attempt.data.get_int('clear_type') in [
+                DBConstants.SDVX_CLEAR_TYPE_ULTIMATE_CHAIN,
+                DBConstants.SDVX_CLEAR_TYPE_PERFECT_ULTIMATE_CHAIN,
+            ]
+
+        return False
+
+    def __aggregate_global(self, attempts: List[Attempt]) -> List[Dict[str, Any]]:
+        stats: Dict[int, Dict[int, Dict[str, int]]] = {}
+
+        for attempt in attempts:
+            if attempt.id not in stats:
+                stats[attempt.id] = {}
+            if attempt.chart not in stats[attempt.id]:
+                stats[attempt.id][attempt.chart] = {
+                    'plays': 0,
+                    'clears': 0,
+                    'combos': 0,
+                }
+
+            if self.__is_play(attempt):
+                stats[attempt.id][attempt.chart]['plays'] += 1
+            if self.__is_clear(attempt):
+                stats[attempt.id][attempt.chart]['clears'] += 1
+            if self.__is_combo(attempt):
+                stats[attempt.id][attempt.chart]['combos'] += 1
+
+        retval = []
+        for songid in stats:
+            for songchart in stats[songid]:
+                stat = stats[songid][songchart]
+                stat['id'] = songid
+                stat['chart'] = songchart
+                retval.append(self.__format_statistics(stat))
+
+        return retval
+
+    def __aggregate_local(self, cards: Dict[int, List[str]], attempts: List[Tuple[UserID, Attempt]]) -> List[Dict[str, Any]]:
+        stats: Dict[UserID, Dict[int, Dict[int, Dict[str, int]]]] = {}
+
+        for (userid, attempt) in attempts:
+            if userid not in stats:
+                stats[userid] = {}
+            if attempt.id not in stats[userid]:
+                stats[userid][attempt.id] = {}
+            if attempt.chart not in stats[userid][attempt.id]:
+                stats[userid][attempt.id][attempt.chart] = {
+                    'plays': 0,
+                    'clears': 0,
+                    'combos': 0,
+                }
+
+            if self.__is_play(attempt):
+                stats[userid][attempt.id][attempt.chart]['plays'] += 1
+            if self.__is_clear(attempt):
+                stats[userid][attempt.id][attempt.chart]['clears'] += 1
+            if self.__is_combo(attempt):
+                stats[userid][attempt.id][attempt.chart]['combos'] += 1
+
+        retval = []
+        for userid in stats:
+            for songid in stats[userid]:
+                for songchart in stats[userid][songid]:
+                    stat = stats[userid][songid][songchart]
+                    stat['id'] = songid
+                    stat['chart'] = songchart
+                    retval.append(self.__format_user_statistics(cards[userid], stat))
+
+        return retval
+
+    def fetch_v1(self, idtype: str, ids: List[str], params: Dict[str, Any]) -> List[Dict[str, Any]]:
+        retval: List[Dict[str, Any]] = []
+
+        # Fetch the attempts
+        if idtype == APIConstants.ID_TYPE_SERVER:
+            retval = self.__aggregate_global(
+                [attempt[1] for attempt in self.data.local.music.get_all_attempts(self.game, self.music_version)]
+            )
+        elif idtype == APIConstants.ID_TYPE_SONG:
+            if len(ids) == 1:
+                songid = int(ids[0])
+                chart = None
+            else:
+                songid = int(ids[0])
+                chart = int(ids[1])
+            retval = self.__aggregate_global(
+                [attempt[1] for attempt in self.data.local.music.get_all_attempts(self.game, self.music_version, songid=songid, songchart=chart)]
+            )
+        elif idtype == APIConstants.ID_TYPE_INSTANCE:
+            songid = int(ids[0])
+            chart = int(ids[1])
+            cardid = ids[2]
+            userid = self.data.local.user.from_cardid(cardid)
+            if userid is not None:
+                retval = self.__aggregate_local(
+                    {userid: self.data.local.user.get_cards(userid)},
+                    self.data.local.music.get_all_attempts(self.game, self.music_version, songid=songid, songchart=chart, userid=userid)
+                )
+        elif idtype == APIConstants.ID_TYPE_CARD:
+            id_to_cards: Dict[int, List[str]] = {}
+            attempts: List[Tuple[UserID, Attempt]] = []
+            for cardid in ids:
+                userid = self.data.local.user.from_cardid(cardid)
+                if userid is not None:
+                    # Don't duplicate loads for users with multiple card IDs if multiples
+                    # of those IDs are requested.
+                    if userid in id_to_cards:
+                        continue
+
+                    id_to_cards[userid] = self.data.local.user.get_cards(userid)
+                    attempts.extend(
+                        self.data.local.music.get_all_attempts(self.game, self.music_version, userid=userid)
+                    )
+            retval = self.__aggregate_local(id_to_cards, attempts)
+        else:
+            raise APIException('Invalid ID type!')
+
+        return retval

bemani/backend/__init__.py

@@ -0,0 +1,2 @@
+from bemani.backend.dispatch import Dispatch, UnrecognizedPCBIDException
+from bemani.backend.base import Base

bemani/backend/base.py

@@ -0,0 +1,434 @@
+import traceback
+from typing import Any, Dict, Iterator, List, Optional, Set, Tuple, Type
+
+from bemani.common import Model, ValidatedDict, Time
+from bemani.data import Data, UserID, RemoteUser
+
+
+class ProfileCreationException(Exception):
+    pass
+
+
+class Status:
+    """
+    List of statuses we return to the game for various reasons.
+    """
+    SUCCESS = 0
+    NO_PROFILE = 109
+    NOT_ALLOWED = 110
+    NOT_REGISTERED = 112
+    INVALID_PIN = 116
+
+
+class Factory:
+    """
+    The base class every game factory inherits from. Defines a create method
+    which should return some game class which can handle packets. Game classes
+    inherit from Base, and have handle_<call>_request methods on them that
+    Dispatch will look up in order to handle calls.
+    """
+
+    MANAGED_CLASSES: List[Type["Base"]] = []
+
+    @classmethod
+    def register_all(cls) -> None:
+        """
+        Subclasses of this class should use this function to register themselves
+        with Base, using Base.register(). Factories specify the game code that
+        they support, which Base will use when routing requests.
+        """
+        raise Exception('Override this in subclass!')
+
+    @classmethod
+    def run_scheduled_work(cls, data: Data, config: Dict[str, Any]) -> None:
+        """
+        Subclasses of this class should use this function to run any scheduled
+        work on classes which it is a factory for. This is usually used for
+        out-of-band DB operations such as generating new weekly/daily charts,
+        calculating league scores, etc.
+        """
+        for game in cls.MANAGED_CLASSES:
+            try:
+                events = game.run_scheduled_work(data, config)
+            except Exception:
+                events = []
+                stack = traceback.format_exc()
+                print(stack)
+                data.local.network.put_event(
+                    'exception',
+                    {
+                        'service': 'scheduler',
+                        'traceback': stack,
+                    },
+                )
+            for event in events:
+                data.local.network.put_event(event[0], event[1])
+
+    @classmethod
+    def all_games(cls) -> Iterator[Tuple[str, int, str]]:
+        """
+        Given a particular factory, iterate over all game, version combinations.
+        Useful for loading things from the DB without wanting to hardcode values.
+        """
+        for game in cls.MANAGED_CLASSES:
+            yield (game.game, game.version, game.name)
+
+    @classmethod
+    def all_settings(cls) -> Iterator[Tuple[str, int, Dict[str, Any]]]:
+        """
+        Given a particular factory, iterate over all game, version combinations that
+        have settings and return those settings.
+        """
+        for game in cls.MANAGED_CLASSES:
+            yield (game.game, game.version, game.get_settings())
+
+    @classmethod
+    def create(cls, data: Data, config: Dict[str, Any], model: Model, parentmodel: Optional[Model]=None) -> Optional['Base']:
+        """
+        Given a modelstring and an optional parent model, return an instantiated game class that can handle a packet.
+
+        Parameters:
+            data - A Data singleton for DB access
+            config - Configuration dictionary
+            model - A parsed Model, used by game factories to determine which game class to return
+            parentmodel - The parent model doing the requesting. In some cases, games request an older
+                          version game class to migrate profiles. This presents a problem when they don't
+                          specify version strings, because some game lookups are ambiguous without them.
+                          This allows a factory to determine which game to return based on the parent
+                          requesting model, assuming that we want one version back.
+
+        Returns:
+            A subclass of Base that hopefully has a handle_<call>_request method on it, for the particular
+            call that Dispatch wants to resolve, or None if we can't look up a game.
+        """
+        raise Exception('Override this in subclass!')
+
+
+class Base:
+    """
+    The base class every game class inherits from. Incudes handlers for card management, PASELI, most
+    non-game startup packets, and simple code for loading/storing profiles.
+    """
+
+    __registered_games: Dict[str, Type[Factory]] = {}
+    __registered_handlers: Set[Type[Factory]] = set()
+
+    """
+    Override this in your subclass.
+    """
+    game = 'dummy'
+
+    """
+    Override this in your subclass.
+    """
+    version = 0
+
+    """
+    Override this in your subclass.
+    """
+    name = 'dummy'
+
+    def __init__(self, data: Data, config: Dict[str, Any], model: Model) -> None:
+        self.data = data
+        self.config = config
+        self.model = model
+
+    @classmethod
+    def create(cls, data: Data, config: Dict[str, Any], model: Model, parentmodel: Optional[Model]=None) -> Optional['Base']:
+        """
+        Given a modelstring and an optional parent model, return an instantiated game class that can handle a packet.
+
+        Note that this is provided here as game factories register with Base to advertise that the will
+        handle some model string. This allows game code to ask for other game classes by model only.
+
+        Parameters:
+            data - A Data singleton for DB access
+            config - Configuration dictionary
+            model - A parsed Model, used by game factories to determine which game class to return
+            parentmodel - The parent model doing the requesting. In some cases, games request an older
+                          version game class to migrate profiles. This presents a problem when they don't
+                          specify version strings, because some game lookups are ambiguous without them.
+                          This allows a factory to determine which game to return based on the parent
+                          requesting model, assuming that we want one version back.
+
+        Returns:
+            A subclass of Base that hopefully has a handle_<call>_request method on it, for the particular
+            call that Dispatch wants to resolve, or an instance of Base itself if no game is registered for
+            this model. Its possible to return None from this function if a registered game has no way of
+            handling this particular modelstring.
+        """
+        if model.game not in cls.__registered_games:
+            # Return just this base model, which will provide nothing
+            return Base(data, config, model)
+        else:
+            # Return the registered module providing this game
+            return cls.__registered_games[model.game].create(data, config, model, parentmodel=parentmodel)
+
+    @classmethod
+    def register(cls, game: str, handler: Type[Factory]) -> None:
+        """
+        Register a factory to handle a game. Note that the game should be the game
+        code as returned by a game, such as "LDJ" or "MDX".
+
+        Parameters:
+            game - 3-character string identifying a game
+            handler - A factory which has a create() method that can spawn game classes.
+        """
+        cls.__registered_games[game] = handler
+        cls.__registered_handlers.add(handler)
+
+    @classmethod
+    def run_scheduled_work(cls, data: Data, config: Dict[str, Any]) -> List[Tuple[str, Dict[str, Any]]]:
+        """
+        Run any out-of-band scheduled work that is applicable to this game.
+        """
+        return []
+
+    @classmethod
+    def get_settings(cls) -> Dict[str, Any]:
+        """
+        Return any game settings this game wishes a front-end to modify.
+        """
+        return {}
+
+    @classmethod
+    def all_games(cls) -> Iterator[Tuple[str, int, str]]:
+        """
+        Given all registered factories, iterate over all game, version combinations.
+        Useful for loading things from the DB without wanting to hardcode values.
+        """
+        for factory in cls.__registered_handlers:
+            for game in factory.MANAGED_CLASSES:
+                yield (game.game, game.version, game.name)
+
+    @classmethod
+    def all_settings(cls) -> Iterator[Tuple[str, int, Dict[str, Any]]]:
+        """
+        Given all registered factories, iterate over all game, version combinations that
+        have settings and return those settings.
+        """
+        for factory in cls.__registered_handlers:
+            for game in factory.MANAGED_CLASSES:
+                yield (game.game, game.version, game.get_settings())
+
+    def extra_services(self) -> List[str]:
+        """
+        A list of extra services that this game needs to advertise.
+        Override in your subclass if you need to advertise extra
+        services for a particular game or series.
+        """
+        return []
+
+    def supports_paseli(self) -> bool:
+        """
+        An override so that particular games can disable PASELI support
+        regardless of the server settings. Some games and some regions
+        are buggy with respect to PASELI.
+        """
+        return True
+
+    def bind_profile(self, userid: UserID) -> None:
+        """
+        Handling binding the user's profile to this version on this server.
+
+        Parameters:
+            userid - The user ID we are binding the profile for.
+        """
+
+    def has_profile(self, userid: UserID) -> bool:
+        """
+        Return whether a user has a profile for this game/version on this server.
+
+        Parameters:
+            userid - The user ID we are binding the profile for.
+
+        Returns:
+            True if the profile exists, False if not.
+        """
+        return self.data.local.user.get_profile(self.game, self.version, userid) is not None
+
+    def get_profile(self, userid: UserID) -> Optional[ValidatedDict]:
+        """
+        Return the profile for a user given this game/version on any connected server.
+
+        Parameters:
+            userid - The user ID we are getting the profile for.
+
+        Returns:
+            A dictionary representing the user's profile, or None if it doesn't exist.
+        """
+        return self.data.remote.user.get_profile(self.game, self.version, userid)
+
+    def get_any_profile(self, userid: UserID) -> ValidatedDict:
+        """
+        Return ANY profile for a user in a game series.
+
+        Tries to look up the profile for a userid/game/version on any connected server.
+        If that fails, looks for the latest profile that the user has for the current
+        game series. This is usually used for fetching profiles to display names for
+        scores, as users can earn scores on different mixes of games and on remote
+        networks.
+
+        Parameters:
+            userid - The user ID we are getting the profile for.
+
+        Returns:
+            A dictionary representing the user's profile, or an empty dictionary if
+            none was found.
+        """
+        profile = self.data.remote.user.get_any_profile(self.game, self.version, userid)
+        if profile is None:
+            profile = ValidatedDict()
+        return profile
+
+    def get_any_profiles(self, userids: List[UserID]) -> List[Tuple[UserID, ValidatedDict]]:
+        """
+        Does the identical thing to the above function, but takes a list of user IDs to
+        fetch in bulk.
+
+        Parameters:
+            userids - List of user IDs we are getting the profile for.
+
+        Returns:
+            A list of tuples with the User ID and dictionary representing the user's profile,
+            or an empty dictionary if nothing was found.
+        """
+        userids = list(set(userids))
+        profiles = self.data.remote.user.get_any_profiles(self.game, self.version, userids)
+        return [
+            (userid, profile if profile is not None else ValidatedDict())
+            for (userid, profile) in profiles
+        ]
+
+    def put_profile(self, userid: UserID, profile: ValidatedDict) -> None:
+        """
+        Save a new profile for this user given a game/version.
+
+        Parameters:
+            userid - The user ID we are saving the profile for.
+            profile - A dictionary that should be looked up later using get_profile.
+        """
+        if RemoteUser.is_remote(userid):
+            raise Exception('Trying to save a remote profile locally!')
+        self.data.local.user.put_profile(self.game, self.version, userid, profile)
+
+    def update_play_statistics(self, userid: UserID, extra_stats: Optional[Dict[str, Any]]=None) -> None:
+        """
+        Given a user ID, calculate new play statistics.
+
+        Handles keeping track of statistics such as consecutive days played, last
+        play date, times played today, times played total, etc.
+
+        Parameters:
+            userid - The user ID we are binding the profile for.
+        """
+        if RemoteUser.is_remote(userid):
+            raise Exception('Trying to save remote statistics locally!')
+
+        # We store the play statistics in a series-wide settings blob so its available
+        # across all game versions, since it isn't game-specific.
+        settings = self.get_play_statistics(userid)
+
+        if extra_stats is not None:
+            for key in extra_stats:
+                # Make sure we don't override anything we manage here
+                if key in [
+                    'total_plays',
+                    'today_plays',
+                    'total_days',
+                    'first_play_timestamp',
+                    'last_play_timestamp',
+                    'last_play_date',
+                    'consecutive_days',
+                ]:
+                    continue
+                # Safe to copy over
+                settings[key] = extra_stats[key]
+
+        settings.replace_int('total_plays', settings.get_int('total_plays') + 1)
+        settings.replace_int('first_play_timestamp', settings.get_int('first_play_timestamp', int(Time.now())))
+        settings.replace_int('last_play_timestamp', int(Time.now()))
+
+        last_play_date = settings.get_int_array('last_play_date', 3)
+        today_play_date = Time.todays_date()
+        yesterday_play_date = Time.yesterdays_date()
+        if (
+            last_play_date[0] == today_play_date[0] and
+            last_play_date[1] == today_play_date[1] and
+            last_play_date[2] == today_play_date[2]
+        ):
+            # We already played today, add one
+            settings.replace_int('today_plays', settings.get_int('today_plays') + 1)
+        else:
+            # We played on a new day, so count total days up
+            settings.replace_int('total_days', settings.get_int('total_days') + 1)
+
+            # We haven't played yet today, reset to one
+            settings.replace_int('today_plays', 1)
+            if (
+                last_play_date[0] == yesterday_play_date[0] and
+                last_play_date[1] == yesterday_play_date[1] and
+                last_play_date[2] == yesterday_play_date[2]
+            ):
+                # We played yesterday, add one to consecutive days
+                settings.replace_int('consecutive_days', settings.get_int('consecutive_days') + 1)
+            else:
+                # We haven't played yet today or yesterday, reset consecutive days
+                settings.replace_int('consecutive_days', 1)
+        settings.replace_int_array('last_play_date', 3, today_play_date)
+
+        # Save back
+        self.data.local.game.put_settings(self.game, userid, settings)
+
+    def get_machine_id(self) -> int:
+        machine = self.data.local.machine.get_machine(self.config['machine']['pcbid'])
+        return machine.id
+
+    def update_machine_name(self, newname: Optional[str]) -> None:
+        if newname is None:
+            return
+        machine = self.data.local.machine.get_machine(self.config['machine']['pcbid'])
+        machine.name = newname
+        self.data.local.machine.put_machine(machine)
+
+    def update_machine_data(self, newdata: Dict[str, Any]) -> None:
+        machine = self.data.local.machine.get_machine(self.config['machine']['pcbid'])
+        machine.data.update(newdata)
+        self.data.local.machine.put_machine(machine)
+
+    def get_game_config(self) -> ValidatedDict:
+        machine = self.data.local.machine.get_machine(self.config['machine']['pcbid'])
+        if machine.arcade is not None:
+            settings = self.data.local.machine.get_settings(machine.arcade, self.game, self.version, 'game_config')
+        else:
+            settings = None
+
+        if settings is None:
+            settings = ValidatedDict()
+        return settings
+
+    def get_play_statistics(self, userid: UserID) -> ValidatedDict:
+        """
+        Given a user ID, get the play statistics.
+
+        Note that games wishing to use this when generating profiles to send to
+        a game should call update_play_statistics when parsing a profile save.
+
+        Parameters:
+            userid - The user ID we are binding the profile for.
+
+        Returns a dictionary optionally containing the following attributes:
+            total_plays - Integer count of total plays for this game series
+            first_play_timestamp - Unix timestamp of first play time
+            last_play_timestamp - Unix timestamp of last play time
+            last_play_date - List of ints in the form of [YYYY, MM, DD] of last play date
+            today_plays - Number of times played today
+            total_days - Total individual days played
+            consecutive_days - Number of consecutive days played at this time.
+        """
+        if RemoteUser.is_remote(userid):
+            return ValidatedDict({})
+        settings = self.data.local.game.get_settings(self.game, userid)
+        if settings is None:
+            return ValidatedDict({})
+        return settings

bemani/backend/bishi/__init__.py

@@ -0,0 +1,2 @@
+from bemani.backend.bishi.factory import BishiBashiFactory
+from bemani.backend.bishi.base import BishiBashiBase

bemani/backend/bishi/base.py

@@ -0,0 +1,23 @@
+# vim: set fileencoding=utf-8
+from typing import Optional
+
+from bemani.backend.base import Base
+from bemani.backend.core import CoreHandler, CardManagerHandler, PASELIHandler
+from bemani.common import GameConstants
+
+
+class BishiBashiBase(CoreHandler, CardManagerHandler, PASELIHandler, Base):
+    """
+    Base game class for all one Bishi Bashi version that we support (lol).
+    In theory we could add support for Bishi Bashi Channel, but that never
+    happened.
+    """
+
+    game = GameConstants.BISHI_BASHI
+
+    def previous_version(self) -> Optional['BishiBashiBase']:
+        """
+        Returns the previous version of the game, based on this game. Should
+        be overridden.
+        """
+        return None

bemani/backend/bishi/bishi.py

@@ -0,0 +1,208 @@
+# vim: set fileencoding=utf-8
+import binascii
+import copy
+import base64
+from typing import Any, Dict, List
+
+from bemani.backend.bishi.base import BishiBashiBase
+from bemani.backend.ess import EventLogHandler
+from bemani.common import ValidatedDict, GameConstants, VersionConstants
+from bemani.data import UserID
+from bemani.protocol import Node
+
+
+class TheStarBishiBashi(
+    EventLogHandler,
+    BishiBashiBase,
+):
+
+    name = "The★BishiBashi"
+    version = VersionConstants.BISHI_BASHI_TSBB
+
+    @classmethod
+    def get_settings(cls) -> Dict[str, Any]:
+        """
+        Return all of our front-end modifiably settings.
+        """
+        return {
+            'bools': [
+                {
+                    'name': 'Force Unlock Characters',
+                    'tip': 'Force unlock all characters on select screen.',
+                    'category': 'game_config',
+                    'setting': 'force_unlock_characters',
+                },
+            ],
+        }
+
+    def __update_shop_name(self, profiledata: bytes) -> None:
+        # Figure out the profile type
+        csvs = profiledata.split(b',')
+        if len(csvs) < 2:
+            # Not long enough to care about
+            return
+        datatype = csvs[1].decode('ascii')
+        if datatype != 'IBBDAT00':
+            # Not the right profile type requested
+            return
+
+        # Grab the shop name
+        try:
+            shopname = csvs[30].decode('shift-jis')
+        except Exception:
+            return
+        self.update_machine_name(shopname)
+
+    def handle_system_getmaster_request(self, request: Node) -> Node:
+        # System message
+        root = Node.void('system')
+        root.add_child(Node.s32('result', 0))
+        return root
+
+    def handle_playerdata_usergamedata_send_request(self, request: Node) -> Node:
+        # Look up user by refid
+        refid = request.child_value('data/eaid')
+        userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        if userid is None:
+            root = Node.void('playerdata')
+            root.add_child(Node.s32('result', 1))  # Unclear if this is the right thing to do here.
+            return root
+
+        # Extract new profile info from old profile
+        oldprofile = self.get_profile(userid)
+        is_new = False
+        if oldprofile is None:
+            oldprofile = ValidatedDict()
+            is_new = True
+        newprofile = self.unformat_profile(userid, request, oldprofile, is_new)
+
+        # Write new profile
+        self.put_profile(userid, newprofile)
+
+        # Return success!
+        root = Node.void('playerdata')
+        root.add_child(Node.s32('result', 0))
+        return root
+
+    def handle_playerdata_usergamedata_recv_request(self, request: Node) -> Node:
+        # Look up user by refid
+        refid = request.child_value('data/eaid')
+        profiletype = request.child_value('data/recv_csv').split(',')[0]
+        profile = None
+        userid = None
+        if refid is not None:
+            userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        if userid is not None:
+            profile = self.get_profile(userid)
+        if profile is not None:
+            return self.format_profile(userid, profiletype, profile)
+        else:
+            root = Node.void('playerdata')
+            root.add_child(Node.s32('result', 1))  # Unclear if this is the right thing to do here.
+            return root
+
+    def format_profile(self, userid: UserID, profiletype: str, profile: ValidatedDict) -> Node:
+        root = Node.void('playerdata')
+        root.add_child(Node.s32('result', 0))
+        player = Node.void('player')
+        root.add_child(player)
+        records = 0
+
+        for i in range(len(profile['strdatas'])):
+            strdata = profile['strdatas'][i]
+            bindata = profile['bindatas'][i]
+
+            # Figure out the profile type
+            csvs = strdata.split(b',')
+            if len(csvs) < 2:
+                # Not long enough to care about
+                continue
+            datatype = csvs[1].decode('ascii')
+            if datatype != profiletype:
+                # Not the right profile type requested
+                continue
+
+            game_config = self.get_game_config()
+            force_unlock_characters = game_config.get_bool('force_unlock_characters')
+            if force_unlock_characters:
+                csvs[11] = b'3ffffffffffff'
+            else:
+                # Reward characters based on playing other games on the network
+                hexdata = csvs[11].decode('ascii')
+                while (len(hexdata) & 1) != 0:
+                    hexdata = '0' + hexdata
+                unlock_bits = [b for b in binascii.unhexlify(hexdata)]
+                while len(unlock_bits) < 7:
+                    unlock_bits.insert(0, 0)
+
+                # Reverse the array, so indexing makes more sense
+                unlock_bits = unlock_bits[::-1]
+
+                # Figure out what other games were played by this user
+                profiles = self.data.local.user.get_games_played(userid)
+
+                # IIDX
+                if len([p for p in profiles if p[0] == GameConstants.IIDX]) > 0:
+                    unlock_bits[1] = unlock_bits[1] | 0x10
+
+                # Pop'n
+                if len([p for p in profiles if p[0] == GameConstants.POPN_MUSIC]) > 0:
+                    unlock_bits[1] = unlock_bits[1] | 0x60
+
+                # Jubeat
+                if len([p for p in profiles if p[0] == GameConstants.JUBEAT]) > 0:
+                    unlock_bits[2] = unlock_bits[2] | 0x02
+
+                # DDR
+                if len([p for p in profiles if p[0] == GameConstants.DDR]) > 0:
+                    unlock_bits[6] = unlock_bits[6] | 0x03
+
+                # GFDM characters exist, but this network has no support for
+                # GFDM or Gitadora, so the bits were never added.
+
+                # Reconstruct table
+                unlock_bits = unlock_bits[::-1]
+                csvs[11] = ''.join(['{:02x}'.format(x) for x in unlock_bits]).encode('ascii')
+
+            # This is a valid profile node for this type, lets return only the profile values
+            strdata = b','.join(csvs[2:])
+            record = Node.void('record')
+            player.add_child(record)
+            d = Node.string('d', base64.b64encode(strdata).decode('ascii'))
+            record.add_child(d)
+            d.add_child(Node.string('bin1', base64.b64encode(bindata).decode('ascii')))
+
+            # Remember that we had this record
+            records = records + 1
+
+        player.add_child(Node.u32('record_num', records))
+        return root
+
+    def unformat_profile(self, userid: UserID, request: Node, oldprofile: ValidatedDict, is_new: bool) -> ValidatedDict:
+        # Profile save request, data values are base64 encoded.
+        # d is a CSV, and bin1 is binary data.
+        newprofile = copy.deepcopy(oldprofile)
+        strdatas: List[bytes] = []
+        bindatas: List[bytes] = []
+
+        record = request.child('data/record')
+        for node in record.children:
+            if node.name != 'd':
+                continue
+
+            profile = base64.b64decode(node.value)
+            # Update the shop name if this is a new profile, since we know it came
+            # from this cabinet. This is the only source of truth for what the
+            # cabinet shop name is set to.
+            if is_new:
+                self.__update_shop_name(profile)
+            strdatas.append(profile)
+            bindatas.append(base64.b64decode(node.child_value('bin1')))
+
+        newprofile['strdatas'] = strdatas
+        newprofile['bindatas'] = bindatas
+
+        # Keep track of play statistics across all versions
+        self.update_play_statistics(userid)
+
+        return newprofile

bemani/backend/bishi/factory.py

@@ -0,0 +1,27 @@
+from typing import Dict, Optional, Any
+
+from bemani.backend.base import Base, Factory
+from bemani.backend.bishi.bishi import TheStarBishiBashi
+from bemani.common import Model
+from bemani.data import Data
+
+
+class BishiBashiFactory(Factory):
+
+    MANAGED_CLASSES = [
+        TheStarBishiBashi,
+    ]
+
+    @classmethod
+    def register_all(cls) -> None:
+        for game in ['IBB']:
+            Base.register(game, BishiBashiFactory)
+
+    @classmethod
+    def create(cls, data: Data, config: Dict[str, Any], model: Model, parentmodel: Optional[Model]=None) -> Optional[Base]:
+
+        if model.game == 'IBB':
+            return TheStarBishiBashi(data, config, model)
+
+        # Unknown game version
+        return None

bemani/backend/core/__init__.py

@@ -0,0 +1,3 @@
+from bemani.backend.core.core import CoreHandler
+from bemani.backend.core.cardmng import CardManagerHandler
+from bemani.backend.core.eacoin import PASELIHandler

bemani/backend/core/cardmng.py

@@ -0,0 +1,116 @@
+from typing import Optional
+
+from bemani.backend.base import Base, Status
+from bemani.protocol import Node
+from bemani.common import Model
+
+
+class CardManagerHandler(Base):
+    """
+    The class that handles card management. This assumes it is attached as a mixin to a game
+    class so that it can understand if there's a profile for a game or not.
+    """
+
+    def handle_cardmng_request(self, request: Node) -> Optional[Node]:
+        """
+        Handle a request for card management. This is independent of a game's profile handling,
+        but still gives the game information as to whether or not a profile exists for a game.
+        These methods handle looking up a card, handling binding a profile to a game version,
+        returning whether a game profile exists or should be migrated, and creating a new account
+        when no account is associated with a card.
+        """
+        method = request.attribute('method')
+
+        if method == 'inquire':
+            # Given a cardid, look up the dataid/refid (same thing in this system).
+            # If the card doesn't exist or isn't allowed, return a status specifying this
+            # instead of the results of the dataid/refid lookup.
+            cardid = request.attribute('cardid')
+            modelstring = request.attribute('model')
+            userid = self.data.local.user.from_cardid(cardid)
+
+            if userid is None:
+                # This user doesn't exist, force system to create new account
+                root = Node.void('cardmng')
+                root.set_attribute('status', str(Status.NOT_REGISTERED))
+                return root
+
+            # Special handling for looking up whether the previous game's profile existed
+            bound = self.has_profile(userid)
+            expired = False
+            if bound is False:
+                if modelstring is not None:
+                    model = Model.from_modelstring(modelstring)
+                    oldgame = Base.create(self.data, self.config, model, self.model)
+                    if oldgame is None:
+                        bound = False
+                    else:
+                        bound = oldgame.has_profile(userid)
+                        expired = True
+
+            refid = self.data.local.user.get_refid(self.game, self.version, userid)
+            paseli_enabled = self.supports_paseli() and self.config['paseli']['enabled']
+
+            root = Node.void('cardmng')
+            root.set_attribute('refid', refid)
+            root.set_attribute('dataid', refid)
+            root.set_attribute('newflag', '1')  # Always seems to be set to 1
+            root.set_attribute('binded', '1' if bound else '0')  # Whether we've bound to this version of the game or not
+            root.set_attribute('expired', '1' if expired else '0')  # Whether we're expired
+            root.set_attribute('ecflag', '1' if paseli_enabled else '0')  # Whether to allow paseli
+            root.set_attribute('useridflag', '1')
+            root.set_attribute('extidflag', '1')
+            return root
+
+        elif method == 'authpass':
+            # Given a dataid/refid previously found via inquire, verify the pin
+            refid = request.attribute('refid')
+            pin = request.attribute('pass')
+            userid = self.data.local.user.from_refid(self.game, self.version, refid)
+            if userid is not None:
+                valid = self.data.local.user.validate_pin(userid, pin)
+            else:
+                valid = False
+            root = Node.void('cardmng')
+            root.set_attribute('status', str(Status.SUCCESS if valid else Status.INVALID_PIN))
+            return root
+
+        elif method == 'getrefid':
+            # Given a cardid and a pin, register the card with the system and generate a new dataid/refid + extid
+            cardid = request.attribute('cardid')
+            pin = request.attribute('passwd')
+            userid = self.data.local.user.create_account(cardid, pin)
+            if userid is None:
+                # This user can't be created
+                root = Node.void('cardmng')
+                root.set_attribute('status', str(Status.NOT_ALLOWED))
+                return root
+
+            refid = self.data.local.user.create_refid(self.game, self.version, userid)
+            root = Node.void('cardmng')
+            root.set_attribute('dataid', refid)
+            root.set_attribute('refid', refid)
+            return root
+
+        elif method == 'bindmodel':
+            # Given a refid, bind the user's card to the current version of the game
+            refid = request.attribute('refid')
+            userid = self.data.local.user.from_refid(self.game, self.version, refid)
+            self.bind_profile(userid)
+            root = Node.void('cardmng')
+            root.set_attribute('dataid', refid)
+            return root
+
+        elif method == 'getkeepspan':
+            # Unclear what this method does, return an arbitrary span
+            root = Node.void('cardmng')
+            root.set_attribute('keepspan', '30')
+            return root
+
+        elif method == 'getdatalist':
+            # Unclear what this method does, return a dummy response
+            root = Node.void('cardmng')
+            return root
+
+        # Invalid method
+        return None

bemani/backend/core/core.py

@@ -0,0 +1,187 @@
+import socket
+
+from bemani.backend.base import Base
+from bemani.protocol import Node
+from bemani.common import ID
+
+
+class CoreHandler(Base):
+    """
+    Implements the core packets that are shared across all games.
+    """
+
+    def handle_services_get_request(self, request: Node) -> Node:
+        """
+        Handles a game request for services.get. This should return the URL of
+        each server which handles a particular service. For us, this is always
+        our URL since we serve everything.
+        """
+        def item(name: str, url: str) -> Node:
+            node = Node.void('item')
+            node.set_attribute('name', name)
+            node.set_attribute('url', url)
+            return node
+
+        url = '{}://{}:{}/'.format(
+            'https' if self.config['server']['https'] else 'http',
+            self.config['server']['address'],
+            self.config['server']['port'],
+        )
+        root = Node.void('services')
+        root.set_attribute('expire', '600')
+        # This can be set to 'operation', 'debug', 'test', and 'factory'.
+        root.set_attribute('mode', 'operation')
+        root.set_attribute('product_domain', '1')
+
+        root.add_child(item('cardmng', url))
+        root.add_child(item('dlstatus', url))
+        root.add_child(item('eacoin', url))
+        root.add_child(item('facility', url))
+        root.add_child(item('lobby', url))
+        root.add_child(item('local', url))
+        root.add_child(item('message', url))
+        root.add_child(item('package', url))
+        root.add_child(item('pcbevent', url))
+        root.add_child(item('pcbtracker', url))
+        root.add_child(item('pkglist', url))
+        root.add_child(item('posevent', url))
+        for srv in self.extra_services():
+            root.add_child(item(srv, url))
+
+        root.add_child(item('ntp', 'ntp://pool.ntp.org/'))
+        # Look up keepalive override if exists, otherwise use the server address
+        if 'keepalive' in self.config['server']:
+            keepalive = self.config['server']['keepalive']
+        else:
+            keepalive = self.config['server']['address']
+        # Translate to a raw IP because we can't give out a host here
+        keepalive = socket.gethostbyname(keepalive)
+        root.add_child(item(
+            'keepalive',
+            'http://{}/core/keepalive?pa={}&ia={}&ga={}&ma={}&t1=2&t2=10'.format(
+                keepalive,
+                keepalive,
+                keepalive,
+                keepalive,
+                keepalive,
+            ),
+        ))
+        return root
+
+    def handle_pcbtracker_alive_request(self, request: Node) -> Node:
+        """
+        Handle a PCBTracker.alive request. The only method of note is the 'alive' method
+        which returns whether PASELI should be active or not for this session.
+        """
+        # Reports that a machine is booting. Overloaded to enable/disable paseli
+        root = Node.void('pcbtracker')
+        root.set_attribute('ecenable', '1' if (self.supports_paseli() and self.config['paseli']['enabled']) else '0')
+        root.set_attribute('expire', '600')
+        return root
+
+    def handle_pcbevent_put_request(self, request: Node) -> Node:
+        """
+        Handle a PCBEvent request. We do nothing for this aside from logging the event.
+        """
+        for item in request.children:
+            if item.name == 'item':
+                name = item.child_value('name')
+                value = item.child_value('value')
+                timestamp = item.child_value('time')
+                self.data.local.network.put_event(
+                    'pcbevent',
+                    {
+                        'name': name,
+                        'value': value,
+                        'model': str(self.model),
+                        'pcbid': self.config['machine']['pcbid'],
+                        'ip': self.config['client']['address'],
+                    },
+                    timestamp=timestamp,
+                )
+
+        return Node.void('pcbevent')
+
+    def handle_package_list_request(self, request: Node) -> Node:
+        """
+        Handle a Package request. This is for supporting downloading of updates.
+        We don't support this at the moment.
+        """
+        # List all available update packages on the server
+        root = Node.void('package')
+        root.set_attribute('expire', '600')
+        return root
+
+    def handle_message_get_request(self, request: Node) -> Node:
+        """
+        I have absolutely no fucking idea what this does, but it might be for
+        operator messages?
+        """
+        root = Node.void('message')
+        root.set_attribute('expire', '600')
+        return root
+
+    def handle_dlstatus_progress_request(self, request: Node) -> Node:
+        """
+        I have absolutely no fucking idea what this does either, download
+        status reports maybe?
+        """
+        return Node.void('dlstatus')
+
+    def handle_facility_get_request(self, request: Node) -> Node:
+        """
+        Handle a facility request. The only method of note is the 'get' request,
+        which expects to return a bunch of information about the arcade this
+        cabinet is in, as well as some settings for URLs and the name of the cab.
+        """
+        machine = self.data.local.machine.get_machine(self.config['machine']['pcbid'])
+
+        root = Node.void('facility')
+        root.set_attribute('expire', '600')
+        location = Node.void('location')
+        location.add_child(Node.string('id', ID.format_machine_id(machine.id)))
+        location.add_child(Node.string('country', 'US'))
+        location.add_child(Node.string('region', '.'))
+        location.add_child(Node.string('name', machine.name))
+        location.add_child(Node.u8('type', 0))
+
+        line = Node.void('line')
+        line.add_child(Node.string('id', '.'))
+        line.add_child(Node.u8('class', 0))
+
+        portfw = Node.void('portfw')
+        portfw.add_child(Node.ipv4('globalip', self.config['client']['address']))
+        portfw.add_child(Node.u16('globalport', machine.port))
+        portfw.add_child(Node.u16('privateport', machine.port))
+
+        public = Node.void('public')
+        public.add_child(Node.u8('flag', 1))
+        public.add_child(Node.string('name', '.'))
+        public.add_child(Node.string('latitude', '0'))
+        public.add_child(Node.string('longitude', '0'))
+
+        share = Node.void('share')
+        eacoin = Node.void('eacoin')
+        eacoin.add_child(Node.s32('notchamount', 3000))
+        eacoin.add_child(Node.s32('notchcount', 3))
+        eacoin.add_child(Node.s32('supplylimit', 10000))
+
+        eapass = Node.void('eapass')
+        eapass.add_child(Node.u16('valid', 365))
+
+        url = Node.void('url')
+        url.add_child(Node.string('eapass', self.config['server']['uri'] or 'www.ea-pass.konami.net'))
+        url.add_child(Node.string('arcadefan', self.config['server']['uri'] or 'www.konami.jp/am'))
+        url.add_child(Node.string('konaminetdx', self.config['server']['uri'] or 'http://am.573.jp'))
+        url.add_child(Node.string('konamiid', self.config['server']['uri'] or 'https://id.konami.net'))
+        url.add_child(Node.string('eagate', self.config['server']['uri'] or 'http://eagate.573.jp'))
+
+        share.add_child(eacoin)
+        share.add_child(url)
+        share.add_child(eapass)
+        root.add_child(location)
+        root.add_child(line)
+        root.add_child(portfw)
+        root.add_child(public)
+        root.add_child(share)
+        return root

bemani/backend/core/eacoin.py

@@ -0,0 +1,439 @@
+from typing import Optional
+
+from bemani.backend.base import Base, Status
+from bemani.protocol import Node
+from bemani.common import Time, CardCipher
+
+
+class PASELIHandler(Base):
+    """
+    A mixin that can be used to provide PASELI services to a game.
+    """
+
+    INFINITE_PASELI_AMOUNT = 57300
+
+    """
+    Override this in your subclass if the particular game/series
+    needs a different padding amount to display PASELI transactions
+    on the operator menu.
+    """
+    paseli_padding = 1
+
+    def handle_eacoin_request(self, request: Node) -> Optional[Node]:
+        """
+        Handle PASELI requests. The game will check out a session at the beginning
+        of the game, make PASELI purchases against that session, and then close it
+        ad the end of of a game. This handler ensures that this works for all games.
+        """
+        method = request.attribute('method')
+
+        if not self.config['paseli']['enabled']:
+            # Refuse to respond, we don't have PASELI enabled
+            print("PASELI not enabled, ignoring eacoin request")
+            root = Node.void('eacoin')
+            root.set_attribute('status', str(Status.NOT_ALLOWED))
+            return root
+
+        if method == 'checkin':
+            root = Node.void('eacoin')
+            cardid = request.child_value('cardid')
+            pin = request.child_value('passwd')
+
+            if cardid is None or pin is None:
+                # Refuse to return anything
+                print("Invalid eacoin checkin request, missing cardid or pin")
+                root.set_attribute('status', str(Status.NO_PROFILE))
+                return root
+
+            userid = self.data.local.user.from_cardid(cardid)
+            if userid is None:
+                # Refuse to do anything
+                print("No user for eacoin checkin request")
+                root.set_attribute('status', str(Status.NO_PROFILE))
+                return root
+
+            valid = self.data.local.user.validate_pin(userid, pin)
+            if not valid:
+                # Refuse to do anything
+                print("User entered invalid pin for eacoin checkin request")
+                root.set_attribute('status', str(Status.INVALID_PIN))
+                return root
+
+            session = self.data.local.user.create_session(userid)
+
+            if self.config['paseli']['infinite']:
+                balance = PASELIHandler.INFINITE_PASELI_AMOUNT
+            else:
+                if self.config['machine']['arcade'] is None:
+                    # There's no arcade for this machine, but infinite is not
+                    # enabled, so there's no way to find a balance.
+                    balance = 0
+                else:
+                    balance = self.data.local.user.get_balance(userid, self.config['machine']['arcade'])
+
+            root.add_child(Node.s16('sequence', 0))
+            root.add_child(Node.u8('acstatus', 0))
+            root.add_child(Node.string('acid', 'DUMMY_ID'))
+            root.add_child(Node.string('acname', 'DUMMY_NAME'))
+            root.add_child(Node.s32('balance', balance))
+            root.add_child(Node.string('sessid', session))
+            return root
+
+        if method == 'opcheckin':
+            root = Node.void('eacoin')
+            passwd = request.child_value('passwd')
+
+            if passwd is None:
+                # Refuse to return anything
+                print("Invalid eacoin checkin request, missing passwd")
+                root.set_attribute('status', str(Status.NO_PROFILE))
+                return root
+
+            if self.config['machine']['arcade'] is None:
+                # Machine doesn't belong to an arcade
+                print("Machine doesn't belong to an arcade")
+                root.set_attribute('status', str(Status.NO_PROFILE))
+                return root
+
+            arcade = self.data.local.machine.get_arcade(self.config['machine']['arcade'])
+            if arcade is None:
+                # Refuse to do anything
+                print("No arcade for operator checkin request")
+                root.set_attribute('status', str(Status.NO_PROFILE))
+                return root
+
+            if arcade.pin != passwd:
+                # Refuse to do anything
+                print("User entered invalid pin for operator checkin request")
+                root.set_attribute('status', str(Status.INVALID_PIN))
+                return root
+
+            session = self.data.local.machine.create_session(arcade.id)
+            root.add_child(Node.string('sessid', session))
+            return root
+
+        elif method == 'consume':
+
+            def make_resp(status: int, balance: int) -> Node:
+                root = Node.void('eacoin')
+                root.add_child(Node.u8('acstatus', status))
+                root.add_child(Node.u8('autocharge', 0))
+                root.add_child(Node.s32('balance', balance))
+                return root
+
+            session = request.child_value('sessid')
+            payment = request.child_value('payment')
+            service = request.child_value('service')
+            details = request.child_value('detail')
+            if session is None or payment is None:
+                # Refuse to do anything
+                print("Invalid eacoin consume request, missing sessid or payment")
+                return make_resp(2, 0)
+
+            userid = self.data.local.user.from_session(session)
+            if session is None:
+                # Refuse to do anything
+                print("Invalid session for eacoin consume request")
+                return make_resp(2, 0)
+
+            if self.config['paseli']['infinite']:
+                balance = PASELIHandler.INFINITE_PASELI_AMOUNT - payment
+            else:
+                if self.config['machine']['arcade'] is None:
+                    # There's no arcade for this machine, but infinite is not
+                    # enabled, so there's no way to find a balance, assume failed
+                    # consume payment.
+                    balance = None
+                else:
+                    # Look up the new balance based on this delta. If there isn't enough,
+                    # we will end up returning None here and exit without performing.
+                    balance = self.data.local.user.update_balance(userid, self.config['machine']['arcade'], -payment)
+
+                if balance is None:
+                    print("Not enough balance for eacoin consume request")
+                    return make_resp(1, self.data.local.user.get_balance(userid, self.config['machine']['arcade']))
+                else:
+                    self.data.local.network.put_event(
+                        'paseli_transaction',
+                        {
+                            'delta': -payment,
+                            'balance': balance,
+                            'service': -service,
+                            'reason': details,
+                            'pcbid': self.config['machine']['pcbid'],
+                        },
+                        userid=userid,
+                        arcadeid=self.config['machine']['arcade'],
+                    )
+
+            return make_resp(0, balance)
+
+        elif method == 'getlog':
+            root = Node.void('eacoin')
+            sessid = request.child_value('sessid')
+            logtype = request.child_value('logtype')
+            target = request.child_value('target')
+            limit = request.child_value('perpage')
+            offset = request.child_value('offset')
+
+            # Try to determine whether its a user or an arcade session
+            userid = self.data.local.user.from_session(sessid)
+            if userid is None:
+                arcadeid = self.data.local.machine.from_session(sessid)
+            else:
+                arcadeid = None
+
+            # Bail out if we don't have any idea what session this is
+            if userid is None and arcadeid is None:
+                print("Unable to determine session type")
+                return root
+
+            # If we're a user session, also look up the current arcade
+            # so we display only entries that happened on this arcade.
+            if userid is not None:
+                arcade = self.data.local.machine.get_arcade(self.config['machine']['arcade'])
+                if arcade is None:
+                    print("Machine doesn't belong to an arcade")
+                    return root
+                arcadeid = arcade.id
+
+            # Now, look up all transactions for this specific group
+            events = self.data.local.network.get_events(
+                userid=userid,
+                arcadeid=arcadeid,
+                event='paseli_transaction',
+            )
+
+            # Further filter it down to the current PCBID
+            events = [event for event in events if event.data.get('pcbid') == target]
+
+            # Grab the end of day today as a timestamp
+            end_of_today = Time.end_of_today()
+            time_format = '%Y-%m-%d %H:%M:%S'
+            date_format = '%Y-%m-%d'
+
+            # Set up common structure
+            lognode = Node.void(logtype)
+            topic = Node.void('topic')
+            lognode.add_child(topic)
+            summary = Node.void('summary')
+            lognode.add_child(summary)
+
+            # Display what day we are summed to
+            topic.add_child(Node.string('sumdate', Time.format(Time.now(), date_format)))
+
+            if logtype == 'last7days':
+                # We show today in the today total, last 7 days prior in the week total
+                beginning_of_today = end_of_today - Time.SECONDS_IN_DAY
+                end_of_week = beginning_of_today
+                beginning_of_week = end_of_week - Time.SECONDS_IN_WEEK
+
+                topic.add_child(Node.string('sumfrom', Time.format(beginning_of_week, date_format)))
+                topic.add_child(Node.string('sumto', Time.format(end_of_week, date_format)))
+                today_total = sum([
+                    -event.data.get_int('delta') for event in events
+                    if event.timestamp >= beginning_of_today and event.timestamp < end_of_today
+                ])
+
+                today_total = sum([
+                    -event.data.get_int('delta') for event in events
+                    if event.timestamp >= beginning_of_today and event.timestamp < end_of_today
+                ])
+                week_txns = [
+                    -event.data.get_int('delta') for event in events
+                    if event.timestamp >= beginning_of_week and event.timestamp < end_of_week
+                ]
+                week_total = sum(week_txns)
+                if len(week_txns) > 0:
+                    week_avg = int(sum(week_txns) / len(week_txns))
+                else:
+                    week_avg = 0
+
+                # We display the totals for each day starting with yesterday and up through 7 days prior.
+                # Index starts at 0 = yesterday, 1 = the day before, etc...
+                items = []
+                for days in range(0, 7):
+                    end_of_day = end_of_week - (days * Time.SECONDS_IN_DAY)
+                    start_of_day = end_of_day - Time.SECONDS_IN_DAY
+
+                    items.append(sum([
+                        -event.data.get_int('delta') for event in events
+                        if event.timestamp >= start_of_day and event.timestamp < end_of_day
+                    ]))
+
+                topic.add_child(Node.s32('today', today_total))
+                topic.add_child(Node.s32('average', week_avg))
+                topic.add_child(Node.s32('total', week_total))
+                summary.add_child(Node.s32_array('items', items))
+
+            if logtype == 'last52weeks':
+                # Start one week back, since the operator can look at last7days for newer stuff.
+                beginning_of_today = end_of_today - Time.SECONDS_IN_DAY
+                end_of_52_weeks = beginning_of_today - Time.SECONDS_IN_WEEK
+
+                topic.add_child(Node.string('sumfrom', Time.format(end_of_52_weeks - (52 * Time.SECONDS_IN_WEEK), date_format)))
+                topic.add_child(Node.string('sumto', Time.format(end_of_52_weeks, date_format)))
+
+                # We index backwards, where index 0 = the first week back, 1 = the next week back after that, etc...
+                items = []
+                for weeks in range(0, 52):
+                    end_of_range = end_of_52_weeks - (weeks * Time.SECONDS_IN_WEEK)
+                    beginning_of_range = end_of_range - Time.SECONDS_IN_WEEK
+
+                    items.append(sum([
+                        -event.data.get_int('delta') for event in events
+                        if event.timestamp >= beginning_of_range and event.timestamp < end_of_range
+                    ]))
+
+                summary.add_child(Node.s32_array('items', items))
+
+            if logtype == 'eachday':
+                start_ts = Time.now()
+                end_ts = Time.now()
+                weekdays = [0] * 7
+
+                for event in events:
+                    event_day = Time.days_into_week(event.timestamp)
+                    weekdays[event_day] = weekdays[event_day] - event.data.get_int('delta')
+                    if event.timestamp < start_ts:
+                        start_ts = event.timestamp
+
+                topic.add_child(Node.string('sumfrom', Time.format(start_ts, date_format)))
+                topic.add_child(Node.string('sumto', Time.format(end_ts, date_format)))
+                summary.add_child(Node.s32_array('items', weekdays))
+
+            if logtype == 'eachhour':
+                start_ts = Time.now()
+                end_ts = Time.now()
+                hours = [0] * 24
+
+                for event in events:
+                    event_hour = int((event.timestamp % Time.SECONDS_IN_DAY) / Time.SECONDS_IN_HOUR)
+                    hours[event_hour] = hours[event_hour] - event.data.get_int('delta')
+                    if event.timestamp < start_ts:
+                        start_ts = event.timestamp
+
+                topic.add_child(Node.string('sumfrom', Time.format(start_ts, date_format)))
+                topic.add_child(Node.string('sumto', Time.format(end_ts, date_format)))
+                summary.add_child(Node.s32_array('items', hours))
+
+            if logtype == 'detail':
+                history = Node.void('history')
+                lognode.add_child(history)
+
+                # Respect details paging
+                if offset is not None:
+                    events = events[offset:]
+                if limit is not None:
+                    events = events[:limit]
+
+                # Output the details themselves
+                for event in events:
+                    card_no = ''
+                    if event.userid is not None:
+                        user = self.data.local.user.get_user(event.userid)
+                        if user is not None:
+                            cards = self.data.local.user.get_cards(user.id)
+                            if len(cards) > 0:
+                                card_no = CardCipher.encode(cards[0])
+
+                    item = Node.void('item')
+                    history.add_child(item)
+                    item.add_child(Node.string('date', Time.format(event.timestamp, time_format)))
+                    item.add_child(Node.s32('consume', -event.data.get_int('delta')))
+                    item.add_child(Node.s32('service', -event.data.get_int('service')))
+                    item.add_child(Node.string('cardtype', ''))
+                    item.add_child(Node.string('cardno', ' ' * self.paseli_padding + card_no))
+                    item.add_child(Node.string('title', ''))
+                    item.add_child(Node.string('systemid', ''))
+
+            if logtype == 'lastmonths':
+                year, month, _ = Time.todays_date()
+                this_month = Time.timestamp_from_date(year, month)
+                last_month = Time.timestamp_from_date(year, month - 1)
+                month_before = Time.timestamp_from_date(year, month - 2)
+
+                topic.add_child(Node.string('sumfrom', Time.format(month_before, date_format)))
+                topic.add_child(Node.string('sumto', Time.format(this_month, date_format)))
+
+                for (start, end) in [(month_before, last_month), (last_month, this_month)]:
+                    year, month, _ = Time.date_from_timestamp(start)
+
+                    items = []
+                    for day in range(0, 31):
+                        begin_ts = start + (day * Time.SECONDS_IN_DAY)
+                        end_ts = begin_ts + Time.SECONDS_IN_DAY
+                        if begin_ts >= end:
+                            # Passed the end of this month
+                            items.append(0)
+                        else:
+                            # Sum up all the txns for this day
+                            items.append(sum([
+                                -event.data.get_int('delta') for event in events
+                                if event.timestamp >= begin_ts and event.timestamp < end_ts
+                            ]))
+
+                    item = Node.void('item')
+                    summary.add_child(item)
+                    item.add_child(Node.s32('year', year))
+                    item.add_child(Node.s32('month', month))
+                    item.add_child(Node.s32_array('items', items))
+
+            root.add_child(Node.u8('processing', 0))
+            root.add_child(lognode)
+            return root
+
+        elif method == 'opchpass':
+            root = Node.void('eacoin')
+            oldpass = request.child_value('passwd')
+            newpass = request.child_value('newpasswd')
+
+            if oldpass is None or newpass is None:
+                # Refuse to return anything
+                print("Invalid eacoin pass change request, missing passwd")
+                root.set_attribute('status', str(Status.NO_PROFILE))
+                return root
+
+            if self.config['machine']['arcade'] is None:
+                # Machine doesn't belong to an arcade
+                print("Machine doesn't belong to an arcade")
+                root.set_attribute('status', str(Status.NO_PROFILE))
+                return root
+
+            arcade = self.data.local.machine.get_arcade(self.config['machine']['arcade'])
+            if arcade is None:
+                # Refuse to do anything
+                print("No arcade for operator pass change request")
+                root.set_attribute('status', str(Status.NO_PROFILE))
+                return root
+
+            if arcade.pin != oldpass:
+                # Refuse to do anything
+                print("User entered invalid pin for operator pass change request")
+                root.set_attribute('status', str(Status.INVALID_PIN))
+                return root
+
+            arcade.pin = newpass
+            self.data.local.machine.put_arcade(arcade)
+            return root
+
+        elif method == 'checkout':
+            session = request.child_value('sessid')
+            if session is not None:
+                # Destroy the session so it can't be used for any other purchases
+                self.data.local.user.destroy_session(session)
+
+            root = Node.void('eacoin')
+            return root
+
+        elif method == 'opcheckout':
+            session = request.child_value('sessid')
+            if session is not None:
+                # Destroy the session so it can't be used for any other purchases
+                self.data.local.machine.destroy_session(session)
+
+            root = Node.void('eacoin')
+            return root
+
+        # Invalid method
+        return None

bemani/backend/ddr/__init__.py

@@ -0,0 +1,2 @@
+from bemani.backend.ddr.factory import DDRFactory
+from bemani.backend.ddr.base import DDRBase

bemani/backend/ddr/base.py

@@ -0,0 +1,345 @@
+# vim: set fileencoding=utf-8
+from typing import Optional, Dict, List, Any
+
+from bemani.backend.base import Base
+from bemani.backend.core import CoreHandler, CardManagerHandler, PASELIHandler
+from bemani.common import Model, ValidatedDict, GameConstants, DBConstants, Time
+from bemani.data import Data, Score, UserID, ScoreSaveException
+from bemani.protocol import Node
+
+
+class DDRBase(CoreHandler, CardManagerHandler, PASELIHandler, Base):
+    """
+    Base game class for all DDR versions. Handles common functionality for getting
+    profiles based on refid, creating new profiles, looking up and saving scores.
+    """
+
+    game = GameConstants.DDR
+
+    HALO_NONE = DBConstants.DDR_HALO_NONE
+    HALO_GOOD_FULL_COMBO = DBConstants.DDR_HALO_GOOD_FULL_COMBO
+    HALO_GREAT_FULL_COMBO = DBConstants.DDR_HALO_GREAT_FULL_COMBO
+    HALO_PERFECT_FULL_COMBO = DBConstants.DDR_HALO_PERFECT_FULL_COMBO
+    HALO_MARVELOUS_FULL_COMBO = DBConstants.DDR_HALO_MARVELOUS_FULL_COMBO
+
+    RANK_E = DBConstants.DDR_RANK_E
+    RANK_D = DBConstants.DDR_RANK_D
+    RANK_D_PLUS = DBConstants.DDR_RANK_D_PLUS
+    RANK_C_MINUS = DBConstants.DDR_RANK_C_MINUS
+    RANK_C = DBConstants.DDR_RANK_C
+    RANK_C_PLUS = DBConstants.DDR_RANK_C_PLUS
+    RANK_B_MINUS = DBConstants.DDR_RANK_B_MINUS
+    RANK_B = DBConstants.DDR_RANK_B
+    RANK_B_PLUS = DBConstants.DDR_RANK_B_PLUS
+    RANK_A_MINUS = DBConstants.DDR_RANK_A_MINUS
+    RANK_A = DBConstants.DDR_RANK_A
+    RANK_A_PLUS = DBConstants.DDR_RANK_A_PLUS
+    RANK_AA_MINUS = DBConstants.DDR_RANK_AA_MINUS
+    RANK_AA = DBConstants.DDR_RANK_AA
+    RANK_AA_PLUS = DBConstants.DDR_RANK_AA_PLUS
+    RANK_AAA = DBConstants.DDR_RANK_AAA
+
+    # These constants must agree with read.py for importing charts from the game.
+    CHART_SINGLE_BEGINNER = 0
+    CHART_SINGLE_BASIC = 1
+    CHART_SINGLE_DIFFICULT = 2
+    CHART_SINGLE_EXPERT = 3
+    CHART_SINGLE_CHALLENGE = 4
+    CHART_DOUBLE_BEGINNER = 5
+    CHART_DOUBLE_BASIC = 6
+    CHART_DOUBLE_DIFFICULT = 7
+    CHART_DOUBLE_EXPERT = 8
+    CHART_DOUBLE_CHALLENGE = 9
+
+    def __init__(self, data: Data, config: Dict[str, Any], model: Model) -> None:
+        super().__init__(data, config, model)
+        if model.rev == 'X':
+            self.omnimix = True
+        else:
+            self.omnimix = False
+
+    @property
+    def music_version(self) -> int:
+        if self.omnimix:
+            return DBConstants.OMNIMIX_VERSION_BUMP + self.version
+        return self.version
+
+    def extra_services(self) -> List[str]:
+        """
+        Return the local2 service so that DDR Ace will send certain packets.
+        """
+        return [
+            'local2',
+        ]
+
+    def game_to_db_rank(self, game_rank: int) -> int:
+        """
+        Given a game's rank constant, return the rank as defined above.
+        """
+        raise Exception('Implement in sub-class!')
+
+    def db_to_game_rank(self, db_rank: int) -> int:
+        """
+        Given a rank as defined above, return the game's rank constant.
+        """
+        raise Exception('Implement in sub-class!')
+
+    def game_to_db_chart(self, game_chart: int) -> int:
+        """
+        Given a game's chart for a song, return the chart as defined above.
+        """
+        raise Exception('Implement in sub-class!')
+
+    def db_to_game_chart(self, db_chart: int) -> int:
+        """
+        Given a chart as defined above, return the game's chart constant.
+        """
+        raise Exception('Implement in sub-class!')
+
+    def game_to_db_halo(self, game_halo: int) -> int:
+        """
+        Given a game's halo constant, return the halo as defined above.
+        """
+        raise Exception('Implement in sub-class!')
+
+    def db_to_game_halo(self, db_halo: int) -> int:
+        """
+        Given a halo as defined above, return the game's halo constant.
+        """
+        raise Exception('Implement in sub-class!')
+
+    def previous_version(self) -> Optional['DDRBase']:
+        """
+        Returns the previous version of the game, based on this game. Should
+        be overridden.
+        """
+        return None
+
+    def format_profile(self, userid: UserID, profile: ValidatedDict) -> Node:
+        """
+        Base handler for a profile. Given a userid and a profile dictionary,
+        return a Node representing a profile. Should be overridden.
+        """
+        return Node.void('game')
+
+    def format_scores(self, userid: UserID, profile: ValidatedDict, scores: List[Score]) -> Node:
+        """
+        Base handler for a score list. Given a userid, profile and a score list,
+        return a Node representing a score list. Should be overridden.
+        """
+        return Node.void('game')
+
+    def unformat_profile(self, userid: UserID, request: Node, oldprofile: ValidatedDict) -> ValidatedDict:
+        """
+        Base handler for profile parsing. Given a request and an old profile,
+        return a new profile that's been updated with the contents of the request.
+        Should be overridden.
+        """
+        return oldprofile
+
+    def get_profile_by_refid(self, refid: Optional[str]) -> Optional[Node]:
+        """
+        Given a RefID, return a formatted profile node. Basically every game
+        needs a profile lookup, even if it handles where that happens in
+        a different request. This is provided for code deduplication.
+        """
+        if refid is None:
+            return None
+
+        # First try to load the actual profile
+        userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        profile = self.get_profile(userid)
+        if profile is None:
+            return None
+
+        # Now, return it
+        return self.format_profile(userid, profile)
+
+    def new_profile_by_refid(self, refid: Optional[str], name: Optional[str], area: Optional[int]) -> None:
+        """
+        Given a RefID and a name/area, create a new profile.
+        """
+        if refid is None:
+            return
+        userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        if userid is None:
+            return
+
+        defaultprofile = ValidatedDict({
+            'name': name,
+            'area': area,
+        })
+        self.put_profile(userid, defaultprofile)
+
+    def put_profile_by_refid(self, refid: Optional[str], request: Node) -> None:
+        """
+        Given a RefID and a request node, unformat the profile and save it.
+        """
+        if refid is None:
+            return
+        userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        if userid is None:
+            return
+
+        oldprofile = self.get_profile(userid)
+        newprofile = self.unformat_profile(userid, request, oldprofile)
+        if newprofile is not None:
+            self.put_profile(userid, newprofile)
+
+    def update_score(
+        self,
+        userid: Optional[UserID],
+        songid: int,
+        chart: int,
+        points: int,
+        rank: int,
+        halo: int,
+        combo: int,
+        trace: Optional[List[int]]=None,
+        ghost: Optional[str]=None,
+    ) -> None:
+        """
+        Given various pieces of a score, update the user's high score and score
+        history in a controlled manner, so all games in DDR series can expect
+        the same attributes in a score.
+        """
+        if chart not in [
+            self.CHART_SINGLE_BEGINNER,
+            self.CHART_SINGLE_BASIC,
+            self.CHART_SINGLE_DIFFICULT,
+            self.CHART_SINGLE_EXPERT,
+            self.CHART_SINGLE_CHALLENGE,
+            self.CHART_DOUBLE_BEGINNER,
+            self.CHART_DOUBLE_BASIC,
+            self.CHART_DOUBLE_DIFFICULT,
+            self.CHART_DOUBLE_EXPERT,
+            self.CHART_DOUBLE_CHALLENGE,
+        ]:
+            raise Exception('Invalid chart {}'.format(chart))
+        if halo not in [
+            self.HALO_NONE,
+            self.HALO_GOOD_FULL_COMBO,
+            self.HALO_GREAT_FULL_COMBO,
+            self.HALO_PERFECT_FULL_COMBO,
+            self.HALO_MARVELOUS_FULL_COMBO,
+        ]:
+            raise Exception('Invalid halo {}'.format(halo))
+        if rank not in [
+            self.RANK_E,
+            self.RANK_D,
+            self.RANK_D_PLUS,
+            self.RANK_C_MINUS,
+            self.RANK_C,
+            self.RANK_C_PLUS,
+            self.RANK_B_MINUS,
+            self.RANK_B,
+            self.RANK_B_PLUS,
+            self.RANK_A_MINUS,
+            self.RANK_A,
+            self.RANK_A_PLUS,
+            self.RANK_AA_MINUS,
+            self.RANK_AA,
+            self.RANK_AA_PLUS,
+            self.RANK_AAA,
+        ]:
+            raise Exception('Invalid rank {}'.format(rank))
+
+        if userid is not None:
+            oldscore = self.data.local.music.get_score(
+                self.game,
+                self.music_version,
+                userid,
+                songid,
+                chart,
+            )
+        else:
+            oldscore = None
+
+        # Score history is verbatum, instead of highest score
+        now = Time.now()
+        history = ValidatedDict({})
+        oldpoints = points
+
+        if oldscore is None:
+            # If it is a new score, create a new dictionary to add to
+            scoredata = ValidatedDict({})
+            raised = True
+            highscore = True
+        else:
+            # Set the score to any new record achieved
+            raised = points > oldscore.points
+            highscore = points >= oldscore.points
+            points = max(oldscore.points, points)
+            scoredata = oldscore.data
+
+        # Save combo
+        history.replace_int('combo', combo)
+        scoredata.replace_int('combo', max(scoredata.get_int('combo'), combo))
+
+        # Save halo
+        history.replace_int('halo', halo)
+        scoredata.replace_int('halo', max(scoredata.get_int('halo'), halo))
+
+        # Save rank
+        history.replace_int('rank', rank)
+        scoredata.replace_int('rank', max(scoredata.get_int('rank'), rank))
+
+        # Save ghost steps
+        if trace is not None:
+            history.replace_int_array('trace', len(trace), trace)
+            if raised:
+                scoredata.replace_int_array('trace', len(trace), trace)
+        if ghost is not None:
+            history.replace_str('ghost', ghost)
+            if raised:
+                scoredata.replace_str('ghost', ghost)
+
+        # Look up where this score was earned
+        lid = self.get_machine_id()
+
+        # DDR sometimes happens to send all songs that were played by a player
+        # at the end of the round. It sends timestamps for the songs, but as of
+        # Colette they were identical for each song in the round. So, if a user
+        # plays the same song/chart# more than once in a round, we will end up
+        # failing to store the attempt since we don't allow two of the same
+        # attempt at the same time for the same user and song/chart. So, bump
+        # the timestamp by one second and retry well past the maximum number of
+        # songs.
+        for bump in range(10):
+            timestamp = now + bump
+
+            if userid is not None:
+                # Write the new score back
+                self.data.local.music.put_score(
+                    self.game,
+                    self.music_version,
+                    userid,
+                    songid,
+                    chart,
+                    lid,
+                    points,
+                    scoredata,
+                    highscore,
+                    timestamp=timestamp,
+                )
+
+            try:
+                # Save the history of this score too
+                self.data.local.music.put_attempt(
+                    self.game,
+                    self.music_version,
+                    userid,
+                    songid,
+                    chart,
+                    lid,
+                    oldpoints,
+                    history,
+                    raised,
+                    timestamp=timestamp,
+                )
+            except ScoreSaveException:
+                # Try again one second in the future
+                continue
+
+            # We saved successfully
+            break

bemani/backend/ddr/common.py

@@ -0,0 +1,508 @@
+from typing import Dict, Optional, Tuple
+
+from bemani.backend.ddr.base import DDRBase
+from bemani.common import Time, ValidatedDict, intish
+from bemani.data import Score, UserID
+from bemani.protocol import Node
+
+
+class DDRGameShopHandler(DDRBase):
+
+    def handle_game_shop_request(self, request: Node) -> Node:
+        self.update_machine_name(request.attribute('name'))
+
+        game = Node.void('game')
+        game.set_attribute('stop', '0')
+        return game
+
+
+class DDRGameLogHandler(DDRBase):
+
+    def handle_game_log_request(self, request: Node) -> Node:
+        return Node.void('game')
+
+
+class DDRGameMessageHandler(DDRBase):
+
+    def handle_game_message_request(self, request: Node) -> Node:
+        return Node.void('game')
+
+
+class DDRGameRankingHandler(DDRBase):
+
+    def handle_game_ranking_request(self, request: Node) -> Node:
+        # Ranking request, unknown what its for
+        return Node.void('game')
+
+
+class DDRGameLockHandler(DDRBase):
+
+    def handle_game_lock_request(self, request: Node) -> Node:
+        game = Node.void('game')
+        game.set_attribute('now_login', '0')
+        return game
+
+
+class DDRGameTaxInfoHandler(DDRBase):
+
+    def handle_game_tax_info_request(self, request: Node) -> Node:
+        game = Node.void('game')
+        tax_info = Node.void('tax_info')
+        game.add_child(tax_info)
+        tax_info.set_attribute('tax_phase', '0')
+        return game
+
+
+class DDRGameRecorderHandler(DDRBase):
+
+    def handle_game_recorder_request(self, request: Node) -> Node:
+        return Node.void('game')
+
+
+class DDRGameHiscoreHandler(DDRBase):
+
+    def handle_game_hiscore_request(self, request: Node) -> Node:
+        records = self.data.remote.music.get_all_records(self.game, self.music_version)
+
+        sortedrecords: Dict[int, Dict[int, Tuple[UserID, Score]]] = {}
+        missing_profiles = []
+        for (userid, score) in records:
+            if score.id not in sortedrecords:
+                sortedrecords[score.id] = {}
+            sortedrecords[score.id][score.chart] = (userid, score)
+            missing_profiles.append(userid)
+        users = {userid: profile for (userid, profile) in self.get_any_profiles(missing_profiles)}
+
+        game = Node.void('game')
+        for song in sortedrecords:
+            music = Node.void('music')
+            game.add_child(music)
+            music.set_attribute('reclink_num', str(song))
+
+            for chart in sortedrecords[song]:
+                userid, score = sortedrecords[song][chart]
+                try:
+                    gamechart = self.db_to_game_chart(chart)
+                except KeyError:
+                    # Don't support this chart in this game
+                    continue
+                gamerank = self.db_to_game_rank(score.data.get_int('rank'))
+                combo_type = self.db_to_game_halo(score.data.get_int('halo'))
+
+                typenode = Node.void('type')
+                music.add_child(typenode)
+                typenode.set_attribute('diff', str(gamechart))
+
+                typenode.add_child(Node.string('name', users[userid].get_str('name')))
+                typenode.add_child(Node.u32('score', score.points))
+                typenode.add_child(Node.u16('area', users[userid].get_int('area', 51)))
+                typenode.add_child(Node.u8('rank', gamerank))
+                typenode.add_child(Node.u8('combo_type', combo_type))
+                typenode.add_child(Node.u32('code', users[userid].get_int('extid')))
+
+        return game
+
+
+class DDRGameAreaHiscoreHandler(DDRBase):
+
+    def handle_game_area_hiscore_request(self, request: Node) -> Node:
+        shop_area = int(request.attribute('shop_area'))
+
+        # First, get all users that are in the current shop's area
+        area_users = {
+            uid: prof for (uid, prof) in self.data.local.user.get_all_profiles(self.game, self.version)
+            if prof.get_int('area', 51) == shop_area
+        }
+
+        # Second, look up records belonging only to those users
+        records = self.data.local.music.get_all_records(self.game, self.music_version, userlist=list(area_users.keys()))
+
+        # Now, do the same lazy thing as 'hiscore' because I don't want
+        # to think about how to change this knowing that we only pulled
+        # up area records.
+        area_records: Dict[int, Dict[int, Tuple[UserID, Score]]] = {}
+        for (userid, score) in records:
+            if score.id not in area_records:
+                area_records[score.id] = {}
+            area_records[score.id][score.chart] = (userid, score)
+
+        game = Node.void('game')
+        for song in area_records:
+            music = Node.void('music')
+            game.add_child(music)
+            music.set_attribute('reclink_num', str(song))
+
+            for chart in area_records[song]:
+                userid, score = area_records[song][chart]
+                if area_users[userid].get_int('area', 51) != shop_area:
+                    # Don't return this, this user isn't in this area
+                    continue
+                try:
+                    gamechart = self.db_to_game_chart(chart)
+                except KeyError:
+                    # Don't support this chart in this game
+                    continue
+                gamerank = self.db_to_game_rank(score.data.get_int('rank'))
+                combo_type = self.db_to_game_halo(score.data.get_int('halo'))
+
+                typenode = Node.void('type')
+                music.add_child(typenode)
+                typenode.set_attribute('diff', str(gamechart))
+
+                typenode.add_child(Node.string('name', area_users[userid].get_str('name')))
+                typenode.add_child(Node.u32('score', score.points))
+                typenode.add_child(Node.u16('area', area_users[userid].get_int('area', 51)))
+                typenode.add_child(Node.u8('rank', gamerank))
+                typenode.add_child(Node.u8('combo_type', combo_type))
+                typenode.add_child(Node.u32('code', area_users[userid].get_int('extid')))
+
+        return game
+
+
+class DDRGameScoreHandler(DDRBase):
+
+    def handle_game_score_request(self, request: Node) -> Node:
+        refid = request.attribute('refid')
+        songid = int(request.attribute('mid'))
+        chart = self.game_to_db_chart(int(request.attribute('type')))
+
+        userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        if userid is not None:
+            attempts = self.data.local.music.get_all_attempts(
+                self.game,
+                self.music_version,
+                userid,
+                songid=songid,
+                songchart=chart,
+                limit=5,
+            )
+            recentscores = [attempt.points for (_, attempt) in attempts]
+        else:
+            recentscores = []
+
+        # Always pad to five, so we ensure that we return all the scores
+        while len(recentscores) < 5:
+            recentscores.append(0)
+
+        # Return the most recent five scores
+        game = Node.void('game')
+        for i in range(len(recentscores)):
+            game.set_attribute('sc{}'.format(i + 1), str(recentscores[i]))
+        return game
+
+
+class DDRGameTraceHandler(DDRBase):
+
+    def handle_game_trace_request(self, request: Node) -> Node:
+        extid = int(request.attribute('code'))
+        chart = int(request.attribute('type'))
+        cid = intish(request.attribute('cid'))
+        mid = intish(request.attribute('mid'))
+
+        # Base packet is just game, if we find something we add to it
+        game = Node.void('game')
+
+        # Rival trace loading
+        userid = self.data.remote.user.from_extid(self.game, self.version, extid)
+        if userid is None:
+            # Nothing to load
+            return game
+
+        if mid is not None:
+            # Load trace from song score
+            songscore = self.data.remote.music.get_score(
+                self.game,
+                self.music_version,
+                userid,
+                mid,
+                self.game_to_db_chart(chart),
+            )
+            if songscore is not None and 'trace' in songscore.data:
+                game.add_child(Node.u32('size', len(songscore.data['trace'])))
+                game.add_child(Node.u8_array('trace', songscore.data['trace']))
+
+        elif cid is not None:
+            # Load trace from achievement
+            coursescore = self.data.local.user.get_achievement(
+                self.game,
+                self.version,
+                userid,
+                (cid * 4) + chart,
+                'course',
+            )
+            if coursescore is not None and 'trace' in coursescore:
+                game.add_child(Node.u32('size', len(coursescore['trace'])))
+                game.add_child(Node.u8_array('trace', coursescore['trace']))
+
+        # Nothing found, return empty
+        return game
+
+
+class DDRGameLoadHandler(DDRBase):
+
+    def handle_game_load_request(self, request: Node) -> Node:
+        refid = request.attribute('refid')
+        profile = self.get_profile_by_refid(refid)
+        if profile is not None:
+            return profile
+
+        game = Node.void('game')
+        game.set_attribute('none', '0')
+        return game
+
+
+class DDRGameLoadDailyHandler(DDRBase):
+
+    def handle_game_load_daily_request(self, request: Node) -> Node:
+        extid = intish(request.attribute('code'))
+        refid = request.attribute('refid')
+        game = Node.void('game')
+        profiledict = None
+
+        if extid is not None:
+            # Rival daily loading
+            userid = self.data.remote.user.from_extid(self.game, self.version, extid)
+        else:
+            # Self daily loading
+            userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        if userid is not None:
+            profiledict = self.get_profile(userid)
+
+        if profiledict is not None:
+            play_stats = self.get_play_statistics(userid)
+
+            # Day play counts
+            last_play_date = play_stats.get_int_array('last_play_date', 3)
+            today_play_date = Time.todays_date()
+            if (
+                last_play_date[0] == today_play_date[0] and
+                last_play_date[1] == today_play_date[1] and
+                last_play_date[2] == today_play_date[2]
+            ):
+                today_count = play_stats.get_int('today_plays', 0)
+            else:
+                today_count = 0
+            daycount = Node.void('daycount')
+            game.add_child(daycount)
+            daycount.set_attribute('playcount', str(today_count))
+
+            # Daily combo stuff, unclear how this works
+            dailycombo = Node.void('dailycombo')
+            game.add_child(dailycombo)
+            dailycombo.set_attribute('daily_combo', str(0))
+            dailycombo.set_attribute('daily_combo_lv', str(0))
+
+        return game
+
+
+class DDRGameOldHandler(DDRBase):
+
+    def handle_game_old_request(self, request: Node) -> Node:
+        refid = request.attribute('refid')
+        game = Node.void('game')
+
+        userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        previous_version: Optional[DDRBase] = None
+        oldprofile: Optional[ValidatedDict] = None
+
+        if userid is not None:
+            previous_version = self.previous_version()
+        if previous_version is not None:
+            oldprofile = previous_version.get_profile(userid)
+        if oldprofile is not None:
+            game.set_attribute('name', oldprofile.get_str('name'))
+            game.set_attribute('area', str(oldprofile.get_int('area', 51)))
+        return game
+
+
+class DDRGameNewHandler(DDRBase):
+
+    def handle_game_new_request(self, request: Node) -> Node:
+        refid = request.attribute('refid')
+        area = int(request.attribute('area'))
+        name = request.attribute('name').strip()
+
+        # Create a new profile for this user!
+        self.new_profile_by_refid(refid, name, area)
+
+        # No response needed
+        game = Node.void('game')
+        return game
+
+
+class DDRGameSaveHandler(DDRBase):
+
+    def handle_game_save_request(self, request: Node) -> Node:
+        refid = request.attribute('refid')
+        self.put_profile_by_refid(refid, request)
+
+        # No response needed
+        game = Node.void('game')
+        return game
+
+
+class DDRGameFriendHandler(DDRBase):
+
+    def handle_game_friend_request(self, request: Node) -> Node:
+        extid = intish(request.attribute('code'))
+        userid = None
+        friend = None
+
+        if extid is not None:
+            # Rival score loading
+            userid = self.data.remote.user.from_extid(self.game, self.version, extid)
+        if userid is not None:
+            friend = self.get_profile(userid)
+            play_stats = self.get_play_statistics(userid)
+
+        if friend is None:
+            # Return an empty node to tell the game we don't have a player here
+            game = Node.void('game')
+            return game
+
+        game = Node.void('game')
+        game.set_attribute('data', '1')
+        game.add_child(Node.u32('code', friend.get_int('extid')))
+        game.add_child(Node.string('name', friend.get_str('name')))
+        game.add_child(Node.u8('area', friend.get_int('area', 51)))
+        game.add_child(Node.u32('exp', play_stats.get_int('exp')))
+        game.add_child(Node.u32('star', friend.get_int('star')))
+
+        # Drill rankings
+        if 'title' in friend:
+            title = Node.void('title')
+            game.add_child(title)
+            titledict = friend.get_dict('title')
+            if 't' in titledict:
+                title.set_attribute('t', str(titledict.get_int('t')))
+            if 's' in titledict:
+                title.set_attribute('s', str(titledict.get_int('s')))
+            if 'd' in titledict:
+                title.set_attribute('d', str(titledict.get_int('d')))
+
+        if 'title_gr' in friend:
+            title_gr = Node.void('title_gr')
+            game.add_child(title_gr)
+            title_grdict = friend.get_dict('title_gr')
+            if 't' in title_grdict:
+                title_gr.set_attribute('t', str(title_grdict.get_int('t')))
+            if 's' in title_grdict:
+                title_gr.set_attribute('s', str(title_grdict.get_int('s')))
+            if 'd' in title_grdict:
+                title_gr.set_attribute('d', str(title_grdict.get_int('d')))
+
+        # Groove gauge level-ups
+        gr_s = Node.void('gr_s')
+        game.add_child(gr_s)
+        index = 1
+        for entry in friend.get_int_array('gr_s', 5):
+            gr_s.set_attribute('gr{}'.format(index), str(entry))
+            index = index + 1
+
+        gr_d = Node.void('gr_d')
+        game.add_child(gr_d)
+        index = 1
+        for entry in friend.get_int_array('gr_d', 5):
+            gr_d.set_attribute('gr{}'.format(index), str(entry))
+            index = index + 1
+        return game
+
+
+class DDRGameLoadCourseHandler(DDRBase):
+
+    def handle_game_load_c_request(self, request: Node) -> Node:
+        extid = intish(request.attribute('code'))
+        refid = request.attribute('refid')
+
+        if extid is not None:
+            # Rival score loading
+            userid = self.data.remote.user.from_extid(self.game, self.version, extid)
+        else:
+            # Self score loading
+            userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+
+        coursedata = [0] * 3200
+        if userid is not None:
+            for course in self.data.local.user.get_achievements(self.game, self.version, userid):
+                if course.type != 'course':
+                    continue
+
+                # Grab course ID and chart (kinda pointless because we add it right back up
+                # below, but it is more documented/readable this way.
+                courseid = int(course.id / 4)
+                coursechart = course.id % 4
+
+                # Populate course data
+                index = ((courseid * 4) + coursechart) * 8
+                if index >= 0 and index <= (len(coursedata) - 8):
+                    coursedata[index + 0] = int(course.data.get_int('score') / 10000)
+                    coursedata[index + 1] = course.data.get_int('score') % 10000
+                    coursedata[index + 2] = course.data.get_int('combo')
+                    coursedata[index + 3] = self.db_to_game_rank(course.data.get_int('rank'))
+                    coursedata[index + 5] = course.data.get_int('stage')
+                    coursedata[index + 6] = course.data.get_int('combo_type')
+
+        game = Node.void('game')
+        game.add_child(Node.u16_array('course', coursedata))
+        return game
+
+
+class DDRGameSaveCourseHandler(DDRBase):
+
+    def handle_game_save_c_request(self, request: Node) -> Node:
+        refid = request.attribute('refid')
+        courseid = int(request.attribute('cid'))
+        chart = int(request.attribute('ctype'))
+
+        userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        if userid is not None:
+            # Calculate statistics
+            data = request.child('data')
+            points = int(data.attribute('score'))
+            combo = int(data.attribute('combo'))
+            combo_type = int(data.attribute('combo_type'))
+            stage = int(data.attribute('stage'))
+            rank = self.game_to_db_rank(int(data.attribute('rank')))
+            trace = request.child_value('trace')
+
+            # Grab the old course score
+            oldcourse = self.data.local.user.get_achievement(
+                self.game,
+                self.version,
+                userid,
+                (courseid * 4) + chart,
+                'course',
+            )
+
+            if oldcourse is not None:
+                highscore = points > oldcourse.get_int('score')
+
+                points = max(points, oldcourse.get_int('score'))
+                combo = max(combo, oldcourse.get_int('combo'))
+                stage = max(stage, oldcourse.get_int('stage'))
+                rank = max(rank, oldcourse.get_int('rank'))
+                combo_type = max(combo_type, oldcourse.get_int('combo_type'))
+
+                if not highscore:
+                    # Don't overwrite the ghost for a non-highscore
+                    trace = oldcourse.get_int_array('trace', len(trace))
+
+            self.data.local.user.put_achievement(
+                self.game,
+                self.version,
+                userid,
+                (courseid * 4) + chart,
+                'course',
+                {
+                    'score': points,
+                    'combo': combo,
+                    'stage': stage,
+                    'rank': rank,
+                    'combo_type': combo_type,
+                    'trace': trace,
+                },
+            )
+
+        # No response needed
+        game = Node.void('game')
+        return game

bemani/backend/ddr/ddr2013.py

@@ -0,0 +1,739 @@
+# vim: set fileencoding=utf-8
+import copy
+from typing import Dict, List, Optional
+
+from bemani.backend.ddr.base import DDRBase
+from bemani.backend.ddr.ddrx3 import DDRX3
+from bemani.backend.ddr.common import (
+    DDRGameAreaHiscoreHandler,
+    DDRGameFriendHandler,
+    DDRGameHiscoreHandler,
+    DDRGameLoadCourseHandler,
+    DDRGameLoadDailyHandler,
+    DDRGameLoadHandler,
+    DDRGameLockHandler,
+    DDRGameLogHandler,
+    DDRGameMessageHandler,
+    DDRGameNewHandler,
+    DDRGameOldHandler,
+    DDRGameRankingHandler,
+    DDRGameRecorderHandler,
+    DDRGameSaveCourseHandler,
+    DDRGameSaveHandler,
+    DDRGameScoreHandler,
+    DDRGameShopHandler,
+    DDRGameTaxInfoHandler,
+    DDRGameTraceHandler,
+)
+from bemani.common import VersionConstants, ValidatedDict, Time, intish
+from bemani.data import Score, UserID
+from bemani.protocol import Node
+
+
+class DDR2013(
+    DDRGameAreaHiscoreHandler,
+    DDRGameFriendHandler,
+    DDRGameHiscoreHandler,
+    DDRGameLoadCourseHandler,
+    DDRGameLoadDailyHandler,
+    DDRGameLoadHandler,
+    DDRGameLockHandler,
+    DDRGameLogHandler,
+    DDRGameMessageHandler,
+    DDRGameNewHandler,
+    DDRGameOldHandler,
+    DDRGameRankingHandler,
+    DDRGameRecorderHandler,
+    DDRGameSaveCourseHandler,
+    DDRGameSaveHandler,
+    DDRGameScoreHandler,
+    DDRGameShopHandler,
+    DDRGameTaxInfoHandler,
+    DDRGameTraceHandler,
+    DDRBase,
+):
+
+    name = 'DanceDanceRevolution 2013'
+    version = VersionConstants.DDR_2013
+
+    GAME_STYLE_SINGLE = 0
+    GAME_STYLE_DOUBLE = 1
+    GAME_STYLE_VERSUS = 2
+
+    GAME_RANK_AAA = 1
+    GAME_RANK_AA = 2
+    GAME_RANK_A = 3
+    GAME_RANK_B = 4
+    GAME_RANK_C = 5
+    GAME_RANK_D = 6
+    GAME_RANK_E = 7
+
+    GAME_CHART_SINGLE_BEGINNER = 0
+    GAME_CHART_SINGLE_BASIC = 1
+    GAME_CHART_SINGLE_DIFFICULT = 2
+    GAME_CHART_SINGLE_EXPERT = 3
+    GAME_CHART_SINGLE_CHALLENGE = 4
+    GAME_CHART_DOUBLE_BASIC = 5
+    GAME_CHART_DOUBLE_DIFFICULT = 6
+    GAME_CHART_DOUBLE_EXPERT = 7
+    GAME_CHART_DOUBLE_CHALLENGE = 8
+
+    GAME_HALO_NONE = 0
+    GAME_HALO_GREAT_COMBO = 1
+    GAME_HALO_PERFECT_COMBO = 2
+    GAME_HALO_MARVELOUS_COMBO = 3
+    GAME_HALO_GOOD_COMBO = 4
+
+    GAME_MAX_SONGS = 700
+
+    def previous_version(self) -> Optional[DDRBase]:
+        return DDRX3(self.data, self.config, self.model)
+
+    def game_to_db_rank(self, game_rank: int) -> int:
+        return {
+            self.GAME_RANK_AAA: self.RANK_AAA,
+            self.GAME_RANK_AA: self.RANK_AA,
+            self.GAME_RANK_A: self.RANK_A,
+            self.GAME_RANK_B: self.RANK_B,
+            self.GAME_RANK_C: self.RANK_C,
+            self.GAME_RANK_D: self.RANK_D,
+            self.GAME_RANK_E: self.RANK_E,
+        }[game_rank]
+
+    def db_to_game_rank(self, db_rank: int) -> int:
+        return {
+            self.RANK_AAA: self.GAME_RANK_AAA,
+            self.RANK_AA_PLUS: self.GAME_RANK_AA,
+            self.RANK_AA: self.GAME_RANK_AA,
+            self.RANK_AA_MINUS: self.GAME_RANK_A,
+            self.RANK_A_PLUS: self.GAME_RANK_A,
+            self.RANK_A: self.GAME_RANK_A,
+            self.RANK_A_MINUS: self.GAME_RANK_B,
+            self.RANK_B_PLUS: self.GAME_RANK_B,
+            self.RANK_B: self.GAME_RANK_B,
+            self.RANK_B_MINUS: self.GAME_RANK_C,
+            self.RANK_C_PLUS: self.GAME_RANK_C,
+            self.RANK_C: self.GAME_RANK_C,
+            self.RANK_C_MINUS: self.GAME_RANK_D,
+            self.RANK_D_PLUS: self.GAME_RANK_D,
+            self.RANK_D: self.GAME_RANK_D,
+            self.RANK_E: self.GAME_RANK_E,
+        }[db_rank]
+
+    def game_to_db_chart(self, game_chart: int) -> int:
+        return {
+            self.GAME_CHART_SINGLE_BEGINNER: self.CHART_SINGLE_BEGINNER,
+            self.GAME_CHART_SINGLE_BASIC: self.CHART_SINGLE_BASIC,
+            self.GAME_CHART_SINGLE_DIFFICULT: self.CHART_SINGLE_DIFFICULT,
+            self.GAME_CHART_SINGLE_EXPERT: self.CHART_SINGLE_EXPERT,
+            self.GAME_CHART_SINGLE_CHALLENGE: self.CHART_SINGLE_CHALLENGE,
+            self.GAME_CHART_DOUBLE_BASIC: self.CHART_DOUBLE_BASIC,
+            self.GAME_CHART_DOUBLE_DIFFICULT: self.CHART_DOUBLE_DIFFICULT,
+            self.GAME_CHART_DOUBLE_EXPERT: self.CHART_DOUBLE_EXPERT,
+            self.GAME_CHART_DOUBLE_CHALLENGE: self.CHART_DOUBLE_CHALLENGE,
+        }[game_chart]
+
+    def db_to_game_chart(self, db_chart: int) -> int:
+        return {
+            self.CHART_SINGLE_BEGINNER: self.GAME_CHART_SINGLE_BEGINNER,
+            self.CHART_SINGLE_BASIC: self.GAME_CHART_SINGLE_BASIC,
+            self.CHART_SINGLE_DIFFICULT: self.GAME_CHART_SINGLE_DIFFICULT,
+            self.CHART_SINGLE_EXPERT: self.GAME_CHART_SINGLE_EXPERT,
+            self.CHART_SINGLE_CHALLENGE: self.GAME_CHART_SINGLE_CHALLENGE,
+            self.CHART_DOUBLE_BASIC: self.GAME_CHART_DOUBLE_BASIC,
+            self.CHART_DOUBLE_DIFFICULT: self.GAME_CHART_DOUBLE_DIFFICULT,
+            self.CHART_DOUBLE_EXPERT: self.GAME_CHART_DOUBLE_EXPERT,
+            self.CHART_DOUBLE_CHALLENGE: self.GAME_CHART_DOUBLE_CHALLENGE,
+        }[db_chart]
+
+    def db_to_game_halo(self, db_halo: int) -> int:
+        if db_halo == self.HALO_MARVELOUS_FULL_COMBO:
+            combo_type = self.GAME_HALO_MARVELOUS_COMBO
+        elif db_halo == self.HALO_PERFECT_FULL_COMBO:
+            combo_type = self.GAME_HALO_PERFECT_COMBO
+        elif db_halo == self.HALO_GREAT_FULL_COMBO:
+            combo_type = self.GAME_HALO_GREAT_COMBO
+        elif db_halo == self.HALO_GOOD_FULL_COMBO:
+            combo_type = self.GAME_HALO_GOOD_COMBO
+        else:
+            combo_type = self.GAME_HALO_NONE
+        return combo_type
+
+    def handle_game_common_request(self, request: Node) -> Node:
+        game = Node.void('game')
+        for flagid in range(256):
+            flag = Node.void('flag')
+            game.add_child(flag)
+
+            flag.set_attribute('id', str(flagid))
+            flag.set_attribute('t', '0')
+            flag.set_attribute('s1', '0')
+            flag.set_attribute('s2', '0')
+            flag.set_attribute('area', '51')
+            flag.set_attribute('is_final', '1')
+
+        hit_chart = self.data.local.music.get_hit_chart(self.game, self.music_version, self.GAME_MAX_SONGS)
+        counts_by_reflink = [0] * self.GAME_MAX_SONGS
+        for (reflink, plays) in hit_chart:
+            if reflink >= 0 and reflink < self.GAME_MAX_SONGS:
+                counts_by_reflink[reflink] = plays
+        game.add_child(Node.u32_array('cnt_music', counts_by_reflink))
+
+        return game
+
+    def handle_game_load_m_request(self, request: Node) -> Node:
+        extid = intish(request.attribute('code'))
+        refid = request.attribute('refid')
+
+        if extid is not None:
+            # Rival score loading
+            userid = self.data.remote.user.from_extid(self.game, self.version, extid)
+        else:
+            # Self score loading
+            userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+
+        if userid is not None:
+            scores = self.data.remote.music.get_scores(self.game, self.music_version, userid)
+        else:
+            scores = []
+
+        sortedscores: Dict[int, Dict[int, Score]] = {}
+        for score in scores:
+            if score.id not in sortedscores:
+                sortedscores[score.id] = {}
+            sortedscores[score.id][score.chart] = score
+
+        game = Node.void('game')
+        for song in sortedscores:
+            music = Node.void('music')
+            game.add_child(music)
+            music.set_attribute('reclink', str(song))
+
+            for chart in sortedscores[song]:
+                score = sortedscores[song][chart]
+                try:
+                    gamechart = self.db_to_game_chart(chart)
+                except KeyError:
+                    # Don't support this chart in this game
+                    continue
+                gamerank = self.db_to_game_rank(score.data.get_int('rank'))
+                combo_type = self.db_to_game_halo(score.data.get_int('halo'))
+
+                typenode = Node.void('type')
+                music.add_child(typenode)
+                typenode.set_attribute('diff', str(gamechart))
+
+                typenode.add_child(Node.u32('score', score.points))
+                typenode.add_child(Node.u16('count', score.plays))
+                typenode.add_child(Node.u8('rank', gamerank))
+                typenode.add_child(Node.u8('combo_type', combo_type))
+
+        return game
+
+    def handle_game_save_m_request(self, request: Node) -> Node:
+        refid = request.attribute('refid')
+        songid = int(request.attribute('mid'))
+        chart = self.game_to_db_chart(int(request.attribute('mtype')))
+
+        # Calculate statistics
+        data = request.child('data')
+        points = int(data.attribute('score'))
+        combo = int(data.attribute('combo'))
+        rank = self.game_to_db_rank(int(data.attribute('rank')))
+        if points == 1000000:
+            halo = self.HALO_MARVELOUS_FULL_COMBO
+        elif int(data.attribute('perf_fc')) != 0:
+            halo = self.HALO_PERFECT_FULL_COMBO
+        elif int(data.attribute('great_fc')) != 0:
+            halo = self.HALO_GREAT_FULL_COMBO
+        elif int(data.attribute('good_fc')) != 0:
+            halo = self.HALO_GOOD_FULL_COMBO
+        else:
+            halo = self.HALO_NONE
+        trace = request.child_value('trace')
+
+        # Save the score, regardless of whether we have a refid. If we save
+        # an anonymous score, it only goes into the DB to count against the
+        # number of plays for that song/chart.
+        userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        self.update_score(
+            userid,
+            songid,
+            chart,
+            points,
+            rank,
+            halo,
+            combo,
+            trace,
+        )
+
+        # No response needed
+        game = Node.void('game')
+        return game
+
+    def format_profile(self, userid: UserID, profile: ValidatedDict) -> Node:
+        root = Node.void('game')
+
+        # Look up play stats we bridge to every mix
+        play_stats = self.get_play_statistics(userid)
+
+        # Basic game settings
+        root.add_child(Node.string('seq', ''))
+        root.add_child(Node.u32('code', profile.get_int('extid')))
+        root.add_child(Node.string('name', profile.get_str('name')))
+        root.add_child(Node.u8('area', profile.get_int('area', 51)))
+        root.add_child(Node.u32('cnt_s', play_stats.get_int('single_plays')))
+        root.add_child(Node.u32('cnt_d', play_stats.get_int('double_plays')))
+        root.add_child(Node.u32('cnt_b', play_stats.get_int('battle_plays')))  # This could be wrong, its a guess
+        root.add_child(Node.u32('cnt_m0', play_stats.get_int('cnt_m0')))
+        root.add_child(Node.u32('cnt_m1', play_stats.get_int('cnt_m1')))
+        root.add_child(Node.u32('cnt_m2', play_stats.get_int('cnt_m2')))
+        root.add_child(Node.u32('cnt_m3', play_stats.get_int('cnt_m3')))
+        root.add_child(Node.u32('cnt_m4', play_stats.get_int('cnt_m4')))
+        root.add_child(Node.u32('cnt_m5', play_stats.get_int('cnt_m5')))
+        root.add_child(Node.u32('exp', play_stats.get_int('exp')))
+        root.add_child(Node.u32('exp_o', profile.get_int('exp_o')))
+        root.add_child(Node.u32('star', profile.get_int('star')))
+        root.add_child(Node.u32('star_c', profile.get_int('star_c')))
+        root.add_child(Node.u8('combo', profile.get_int('combo', 0)))
+        root.add_child(Node.u8('timing_diff', profile.get_int('early_late', 0)))
+
+        # Character stuff
+        chara = Node.void('chara')
+        root.add_child(chara)
+        chara.set_attribute('my', str(profile.get_int('chara', 30)))
+        root.add_child(Node.u16_array('chara_opt', profile.get_int_array('chara_opt', 96, [208] * 96)))
+
+        # Drill rankings
+        if 'title_gr' in profile:
+            title_gr = Node.void('title_gr')
+            root.add_child(title_gr)
+            title_grdict = profile.get_dict('title_gr')
+            if 't' in title_grdict:
+                title_gr.set_attribute('t', str(title_grdict.get_int('t')))
+            if 's' in title_grdict:
+                title_gr.set_attribute('s', str(title_grdict.get_int('s')))
+            if 'd' in title_grdict:
+                title_gr.set_attribute('d', str(title_grdict.get_int('d')))
+
+        # Calorie mode
+        if 'weight' in profile:
+            workouts = self.data.local.user.get_time_based_achievements(
+                self.game,
+                self.version,
+                userid,
+                achievementtype='workout',
+                since=Time.now() - Time.SECONDS_IN_DAY,
+            )
+            total = sum([w.data.get_int('calories') for w in workouts])
+            workout = Node.void('workout')
+            root.add_child(workout)
+            workout.set_attribute('weight', str(profile.get_int('weight')))
+            workout.set_attribute('day', str(total))
+            workout.set_attribute('disp', '1')
+
+        # Daily play counts
+        last_play_date = play_stats.get_int_array('last_play_date', 3)
+        today_play_date = Time.todays_date()
+        if (
+            last_play_date[0] == today_play_date[0] and
+            last_play_date[1] == today_play_date[1] and
+            last_play_date[2] == today_play_date[2]
+        ):
+            today_count = play_stats.get_int('today_plays', 0)
+        else:
+            today_count = 0
+        daycount = Node.void('daycount')
+        root.add_child(daycount)
+        daycount.set_attribute('playcount', str(today_count))
+
+        # Daily combo stuff, unknown how this works
+        dailycombo = Node.void('dailycombo')
+        root.add_child(dailycombo)
+        dailycombo.set_attribute('daily_combo', str(0))
+        dailycombo.set_attribute('daily_combo_lv', str(0))
+
+        # Last cursor settings
+        last = Node.void('last')
+        root.add_child(last)
+        lastdict = profile.get_dict('last')
+        last.set_attribute('rival1', str(lastdict.get_int('rival1', -1)))
+        last.set_attribute('rival2', str(lastdict.get_int('rival2', -1)))
+        last.set_attribute('rival3', str(lastdict.get_int('rival3', -1)))
+        last.set_attribute('fri', str(lastdict.get_int('rival1', -1)))  # This literally goes to the same memory in 2013
+        last.set_attribute('style', str(lastdict.get_int('style')))
+        last.set_attribute('mode', str(lastdict.get_int('mode')))
+        last.set_attribute('cate', str(lastdict.get_int('cate')))
+        last.set_attribute('sort', str(lastdict.get_int('sort')))
+        last.set_attribute('mid', str(lastdict.get_int('mid')))
+        last.set_attribute('mtype', str(lastdict.get_int('mtype')))
+        last.set_attribute('cid', str(lastdict.get_int('cid')))
+        last.set_attribute('ctype', str(lastdict.get_int('ctype')))
+        last.set_attribute('sid', str(lastdict.get_int('sid')))
+
+        # Result stars
+        result_star = Node.void('result_star')
+        root.add_child(result_star)
+        result_stars = profile.get_int_array('result_stars', 9)
+        for i in range(9):
+            result_star.set_attribute('slot{}'.format(i + 1), str(result_stars[i]))
+
+        # Target stuff
+        target = Node.void('target')
+        root.add_child(target)
+        target.set_attribute('flag', str(profile.get_int('target_flag')))
+        target.set_attribute('setnum', str(profile.get_int('target_setnum')))
+
+        # Groove gauge level-ups
+        gr_s = Node.void('gr_s')
+        root.add_child(gr_s)
+        index = 1
+        for entry in profile.get_int_array('gr_s', 5):
+            gr_s.set_attribute('gr{}'.format(index), str(entry))
+            index = index + 1
+
+        gr_d = Node.void('gr_d')
+        root.add_child(gr_d)
+        index = 1
+        for entry in profile.get_int_array('gr_d', 5):
+            gr_d.set_attribute('gr{}'.format(index), str(entry))
+            index = index + 1
+
+        # Options in menus
+        root.add_child(Node.s16_array('opt', profile.get_int_array('opt', 16)))
+        root.add_child(Node.s16_array('opt_ex', profile.get_int_array('opt_ex', 16)))
+
+        # Unlock flags
+        root.add_child(Node.u8_array('flag', profile.get_int_array('flag', 256, [1] * 256)))
+
+        # Ranking display?
+        root.add_child(Node.u16_array('rank', profile.get_int_array('rank', 100)))
+
+        # Rivals
+        links = self.data.local.user.get_links(self.game, self.version, userid)
+        for link in links:
+            if link.type[:7] != 'friend_':
+                continue
+
+            pos = int(link.type[7:])
+            friend = self.get_profile(link.other_userid)
+            play_stats = self.get_play_statistics(link.other_userid)
+            if friend is not None:
+                friendnode = Node.void('friend')
+                root.add_child(friendnode)
+                friendnode.set_attribute('pos', str(pos))
+                friendnode.set_attribute('vs', '0')
+                friendnode.set_attribute('up', '0')
+                friendnode.add_child(Node.u32('code', friend.get_int('extid')))
+                friendnode.add_child(Node.string('name', friend.get_str('name')))
+                friendnode.add_child(Node.u8('area', friend.get_int('area', 51)))
+                friendnode.add_child(Node.u32('exp', play_stats.get_int('exp')))
+                friendnode.add_child(Node.u32('star', friend.get_int('star')))
+
+                # Drill rankings
+                if 'title' in friend:
+                    title = Node.void('title')
+                    friendnode.add_child(title)
+                    titledict = friend.get_dict('title')
+                    if 't' in titledict:
+                        title.set_attribute('t', str(titledict.get_int('t')))
+                    if 's' in titledict:
+                        title.set_attribute('s', str(titledict.get_int('s')))
+                    if 'd' in titledict:
+                        title.set_attribute('d', str(titledict.get_int('d')))
+
+                if 'title_gr' in friend:
+                    title_gr = Node.void('title_gr')
+                    friendnode.add_child(title_gr)
+                    title_grdict = friend.get_dict('title_gr')
+                    if 't' in title_grdict:
+                        title_gr.set_attribute('t', str(title_grdict.get_int('t')))
+                    if 's' in title_grdict:
+                        title_gr.set_attribute('s', str(title_grdict.get_int('s')))
+                    if 'd' in title_grdict:
+                        title_gr.set_attribute('d', str(title_grdict.get_int('d')))
+
+                # Groove gauge level-ups
+                gr_s = Node.void('gr_s')
+                friendnode.add_child(gr_s)
+                index = 1
+                for entry in friend.get_int_array('gr_s', 5):
+                    gr_s.set_attribute('gr{}'.format(index), str(entry))
+                    index = index + 1
+
+                gr_d = Node.void('gr_d')
+                friendnode.add_child(gr_d)
+                index = 1
+                for entry in friend.get_int_array('gr_d', 5):
+                    gr_d.set_attribute('gr{}'.format(index), str(entry))
+                    index = index + 1
+
+        # Play area
+        areas = profile.get_int_array('play_area', 55)
+        play_area = Node.void('play_area')
+        root.add_child(play_area)
+        for i in range(len(areas)):
+            play_area.set_attribute('play_cnt{}'.format(i), str(areas[i]))
+
+        return root
+
+    def unformat_profile(self, userid: UserID, request: Node, oldprofile: ValidatedDict) -> ValidatedDict:
+        newprofile = copy.deepcopy(oldprofile)
+        play_stats = self.get_play_statistics(userid)
+
+        # Grab last node and accessories so we can make decisions based on type
+        last = request.child('last')
+        lastdict = newprofile.get_dict('last')
+        mode = int(last.attribute('mode'))
+        style = int(last.attribute('style'))
+        is_dp = style == self.GAME_STYLE_DOUBLE
+
+        # Drill rankings
+        title = request.child('title')
+        title_gr = request.child('title_gr')
+        titledict = newprofile.get_dict('title')
+        title_grdict = newprofile.get_dict('title_gr')
+
+        # Groove radar level ups
+        gr = request.child('gr')
+
+        # Set the correct values depending on if we're single or double play
+        if is_dp:
+            play_stats.increment_int('double_plays')
+            if gr is not None:
+                newprofile.replace_int_array(
+                    'gr_d',
+                    5,
+                    [
+                        intish(gr.attribute('gr1')),
+                        intish(gr.attribute('gr2')),
+                        intish(gr.attribute('gr3')),
+                        intish(gr.attribute('gr4')),
+                        intish(gr.attribute('gr5')),
+                    ],
+                )
+            if title is not None:
+                titledict.replace_int('d', title.value)
+                newprofile.replace_dict('title', titledict)
+            if title_gr is not None:
+                title_grdict.replace_int('d', title.value)
+                newprofile.replace_dict('title_gr', title_grdict)
+        else:
+            play_stats.increment_int('single_plays')
+            if gr is not None:
+                newprofile.replace_int_array(
+                    'gr_s',
+                    5,
+                    [
+                        intish(gr.attribute('gr1')),
+                        intish(gr.attribute('gr2')),
+                        intish(gr.attribute('gr3')),
+                        intish(gr.attribute('gr4')),
+                        intish(gr.attribute('gr5')),
+                    ],
+                )
+            if title is not None:
+                titledict.replace_int('s', title.value)
+                newprofile.replace_dict('title', titledict)
+            if title_gr is not None:
+                title_grdict.replace_int('s', title.value)
+                newprofile.replace_dict('title_gr', title_grdict)
+        play_stats.increment_int('cnt_m{}'.format(mode))
+
+        # Result stars
+        result_star = request.child('result_star')
+        if result_star is not None:
+            newprofile.replace_int_array(
+                'result_stars',
+                9,
+                [
+                    intish(result_star.attribute('slot1')),
+                    intish(result_star.attribute('slot2')),
+                    intish(result_star.attribute('slot3')),
+                    intish(result_star.attribute('slot4')),
+                    intish(result_star.attribute('slot5')),
+                    intish(result_star.attribute('slot6')),
+                    intish(result_star.attribute('slot7')),
+                    intish(result_star.attribute('slot8')),
+                    intish(result_star.attribute('slot9')),
+                ],
+            )
+
+        # Target stuff
+        target = request.child('target')
+        if target is not None:
+            newprofile.replace_int('target_flag', intish(target.attribute('flag')))
+            newprofile.replace_int('target_setnum', intish(target.attribute('setnum')))
+
+        # Update last attributes
+        lastdict.replace_int('rival1', intish(last.attribute('rival1')))
+        lastdict.replace_int('rival2', intish(last.attribute('rival2')))
+        lastdict.replace_int('rival3', intish(last.attribute('rival3')))
+        lastdict.replace_int('style', intish(last.attribute('style')))
+        lastdict.replace_int('mode', intish(last.attribute('mode')))
+        lastdict.replace_int('cate', intish(last.attribute('cate')))
+        lastdict.replace_int('sort', intish(last.attribute('sort')))
+        lastdict.replace_int('mid', intish(last.attribute('mid')))
+        lastdict.replace_int('mtype', intish(last.attribute('mtype')))
+        lastdict.replace_int('cid', intish(last.attribute('cid')))
+        lastdict.replace_int('ctype', intish(last.attribute('ctype')))
+        lastdict.replace_int('sid', intish(last.attribute('sid')))
+        newprofile.replace_dict('last', lastdict)
+
+        # Grab character options
+        chara = request.child('chara')
+        if chara is not None:
+            newprofile.replace_int('chara', intish(chara.attribute('my')))
+        chara_opt = request.child('chara_opt')
+        if chara_opt is not None:
+            # A bug in old versions of AVS returns the wrong number for set
+            newprofile.replace_int_array('chara_opt', 96, chara_opt.value[:96])
+
+        # Options
+        opt = request.child('opt')
+        if opt is not None:
+            # A bug in old versions of AVS returns the wrong number for set
+            newprofile.replace_int_array('opt', 16, opt.value[:16])
+
+        # Experience and stars
+        exp = request.child_value('exp')
+        if exp is not None:
+            play_stats.replace_int('exp', play_stats.get_int('exp') + exp)
+        star = request.child_value('star')
+        if star is not None:
+            newprofile.replace_int('star', newprofile.get_int('star') + star)
+        star_c = request.child_value('star_c')
+        if star_c is not None:
+            newprofile.replace_int('star_c', newprofile.get_int('star_c') + exp)
+
+        # Update game flags
+        for child in request.children:
+            if child.name != 'flag':
+                continue
+            try:
+                value = int(child.attribute('data'))
+                offset = int(child.attribute('no'))
+            except ValueError:
+                continue
+
+            flags = newprofile.get_int_array('flag', 256, [1] * 256)
+            if offset < 0 or offset >= len(flags):
+                continue
+            flags[offset] = value
+            newprofile.replace_int_array('flag', 256, flags)
+
+        # Workout mode support
+        newweight = -1
+        oldweight = newprofile.get_int('weight')
+        for child in request.children:
+            if child.name != 'weight':
+                continue
+            newweight = child.value
+        if newweight < 0:
+            newweight = oldweight
+
+        # Either update or unset the weight depending on the game
+        if newweight == 0:
+            # Weight is unset or we declined to use this feature, remove from profile
+            if 'weight' in newprofile:
+                del newprofile['weight']
+        else:
+            # Weight has been set or previously retrieved, we should save calories
+            newprofile.replace_int('weight', newweight)
+            total = 0
+            for child in request.children:
+                if child.name != 'calory':
+                    continue
+                total += child.value
+            self.data.local.user.put_time_based_achievement(
+                self.game,
+                self.version,
+                userid,
+                0,
+                'workout',
+                {
+                    'calories': total,
+                    'weight': newweight,
+                },
+            )
+
+        # Look up old friends
+        oldfriends: List[Optional[UserID]] = [None] * 10
+        links = self.data.local.user.get_links(self.game, self.version, userid)
+        for link in links:
+            if link.type[:7] != 'friend_':
+                continue
+
+            pos = int(link.type[7:])
+            oldfriends[pos] = link.other_userid
+
+        # Save any rivals that were added/removed/changed
+        newfriends = oldfriends[:]
+        for child in request.children:
+            if child.name != 'friend':
+                continue
+
+            code = int(child.attribute('code'))
+            pos = int(child.attribute('pos'))
+
+            if pos >= 0 and pos < 10:
+                if code == 0:
+                    # We cleared this friend
+                    newfriends[pos] = None
+                else:
+                    # Try looking up the userid
+                    newfriends[pos] = self.data.remote.user.from_extid(self.game, self.version, code)
+
+        # Diff the set of links to determine updates
+        for i in range(10):
+            if newfriends[i] == oldfriends[i]:
+                continue
+
+            if newfriends[i] is None:
+                # Kill the rival in this location
+                self.data.local.user.destroy_link(
+                    self.game,
+                    self.version,
+                    userid,
+                    'friend_{}'.format(i),
+                    oldfriends[i],
+                )
+            elif oldfriends[i] is None:
+                # Add rival in this location
+                self.data.local.user.put_link(
+                    self.game,
+                    self.version,
+                    userid,
+                    'friend_{}'.format(i),
+                    newfriends[i],
+                    {},
+                )
+            else:
+                # Changed the rival here, kill the old one, add the new one
+                self.data.local.user.destroy_link(
+                    self.game,
+                    self.version,
+                    userid,
+                    'friend_{}'.format(i),
+                    oldfriends[i],
+                )
+                self.data.local.user.put_link(
+                    self.game,
+                    self.version,
+                    userid,
+                    'friend_{}'.format(i),
+                    newfriends[i],
+                    {},
+                )
+
+        # Play area counter
+        shop_area = int(request.attribute('shop_area'))
+        if shop_area >= 0 and shop_area < 55:
+            areas = newprofile.get_int_array('play_area', 55)
+            areas[shop_area] = areas[shop_area] + 1
+            newprofile.replace_int_array('play_area', 55, areas)
+
+        # Keep track of play statistics
+        self.update_play_statistics(userid, play_stats)
+
+        return newprofile

bemani/backend/ddr/ddr2014.py

@@ -0,0 +1,800 @@
+# vim: set fileencoding=utf-8
+import copy
+from typing import Dict, List, Optional
+
+from bemani.backend.ddr.base import DDRBase
+from bemani.backend.ddr.ddr2013 import DDR2013
+from bemani.backend.ddr.common import (
+    DDRGameAreaHiscoreHandler,
+    DDRGameFriendHandler,
+    DDRGameHiscoreHandler,
+    DDRGameLoadDailyHandler,
+    DDRGameLoadHandler,
+    DDRGameLockHandler,
+    DDRGameLogHandler,
+    DDRGameMessageHandler,
+    DDRGameNewHandler,
+    DDRGameOldHandler,
+    DDRGameRankingHandler,
+    DDRGameRecorderHandler,
+    DDRGameSaveHandler,
+    DDRGameScoreHandler,
+    DDRGameShopHandler,
+    DDRGameTaxInfoHandler,
+)
+from bemani.common import VersionConstants, ValidatedDict, Time, intish
+from bemani.data import Score, UserID
+from bemani.protocol import Node
+
+
+class DDR2014(
+    DDRGameAreaHiscoreHandler,
+    DDRGameFriendHandler,
+    DDRGameHiscoreHandler,
+    DDRGameLoadDailyHandler,
+    DDRGameLoadHandler,
+    DDRGameLockHandler,
+    DDRGameLogHandler,
+    DDRGameMessageHandler,
+    DDRGameNewHandler,
+    DDRGameOldHandler,
+    DDRGameRankingHandler,
+    DDRGameRecorderHandler,
+    DDRGameSaveHandler,
+    DDRGameScoreHandler,
+    DDRGameShopHandler,
+    DDRGameTaxInfoHandler,
+    DDRBase,
+):
+
+    name = 'DanceDanceRevolution 2014'
+    version = VersionConstants.DDR_2014
+
+    GAME_STYLE_SINGLE = 0
+    GAME_STYLE_DOUBLE = 1
+    GAME_STYLE_VERSUS = 2
+
+    GAME_RANK_AAA = 1
+    GAME_RANK_AA = 2
+    GAME_RANK_A = 3
+    GAME_RANK_B = 4
+    GAME_RANK_C = 5
+    GAME_RANK_D = 6
+    GAME_RANK_E = 7
+
+    GAME_CHART_SINGLE_BEGINNER = 0
+    GAME_CHART_SINGLE_BASIC = 1
+    GAME_CHART_SINGLE_DIFFICULT = 2
+    GAME_CHART_SINGLE_EXPERT = 3
+    GAME_CHART_SINGLE_CHALLENGE = 4
+    GAME_CHART_DOUBLE_BASIC = 5
+    GAME_CHART_DOUBLE_DIFFICULT = 6
+    GAME_CHART_DOUBLE_EXPERT = 7
+    GAME_CHART_DOUBLE_CHALLENGE = 8
+
+    GAME_HALO_NONE = 0
+    GAME_HALO_GREAT_COMBO = 1
+    GAME_HALO_PERFECT_COMBO = 2
+    GAME_HALO_MARVELOUS_COMBO = 3
+    GAME_HALO_GOOD_COMBO = 4
+
+    GAME_MAX_SONGS = 800
+
+    def previous_version(self) -> Optional[DDRBase]:
+        return DDR2013(self.data, self.config, self.model)
+
+    def game_to_db_rank(self, game_rank: int) -> int:
+        return {
+            self.GAME_RANK_AAA: self.RANK_AAA,
+            self.GAME_RANK_AA: self.RANK_AA,
+            self.GAME_RANK_A: self.RANK_A,
+            self.GAME_RANK_B: self.RANK_B,
+            self.GAME_RANK_C: self.RANK_C,
+            self.GAME_RANK_D: self.RANK_D,
+            self.GAME_RANK_E: self.RANK_E,
+        }[game_rank]
+
+    def db_to_game_rank(self, db_rank: int) -> int:
+        return {
+            self.RANK_AAA: self.GAME_RANK_AAA,
+            self.RANK_AA_PLUS: self.GAME_RANK_AA,
+            self.RANK_AA: self.GAME_RANK_AA,
+            self.RANK_AA_MINUS: self.GAME_RANK_A,
+            self.RANK_A_PLUS: self.GAME_RANK_A,
+            self.RANK_A: self.GAME_RANK_A,
+            self.RANK_A_MINUS: self.GAME_RANK_B,
+            self.RANK_B_PLUS: self.GAME_RANK_B,
+            self.RANK_B: self.GAME_RANK_B,
+            self.RANK_B_MINUS: self.GAME_RANK_C,
+            self.RANK_C_PLUS: self.GAME_RANK_C,
+            self.RANK_C: self.GAME_RANK_C,
+            self.RANK_C_MINUS: self.GAME_RANK_D,
+            self.RANK_D_PLUS: self.GAME_RANK_D,
+            self.RANK_D: self.GAME_RANK_D,
+            self.RANK_E: self.GAME_RANK_E,
+        }[db_rank]
+
+    def game_to_db_chart(self, game_chart: int) -> int:
+        return {
+            self.GAME_CHART_SINGLE_BEGINNER: self.CHART_SINGLE_BEGINNER,
+            self.GAME_CHART_SINGLE_BASIC: self.CHART_SINGLE_BASIC,
+            self.GAME_CHART_SINGLE_DIFFICULT: self.CHART_SINGLE_DIFFICULT,
+            self.GAME_CHART_SINGLE_EXPERT: self.CHART_SINGLE_EXPERT,
+            self.GAME_CHART_SINGLE_CHALLENGE: self.CHART_SINGLE_CHALLENGE,
+            self.GAME_CHART_DOUBLE_BASIC: self.CHART_DOUBLE_BASIC,
+            self.GAME_CHART_DOUBLE_DIFFICULT: self.CHART_DOUBLE_DIFFICULT,
+            self.GAME_CHART_DOUBLE_EXPERT: self.CHART_DOUBLE_EXPERT,
+            self.GAME_CHART_DOUBLE_CHALLENGE: self.CHART_DOUBLE_CHALLENGE,
+        }[game_chart]
+
+    def db_to_game_chart(self, db_chart: int) -> int:
+        return {
+            self.CHART_SINGLE_BEGINNER: self.GAME_CHART_SINGLE_BEGINNER,
+            self.CHART_SINGLE_BASIC: self.GAME_CHART_SINGLE_BASIC,
+            self.CHART_SINGLE_DIFFICULT: self.GAME_CHART_SINGLE_DIFFICULT,
+            self.CHART_SINGLE_EXPERT: self.GAME_CHART_SINGLE_EXPERT,
+            self.CHART_SINGLE_CHALLENGE: self.GAME_CHART_SINGLE_CHALLENGE,
+            self.CHART_DOUBLE_BASIC: self.GAME_CHART_DOUBLE_BASIC,
+            self.CHART_DOUBLE_DIFFICULT: self.GAME_CHART_DOUBLE_DIFFICULT,
+            self.CHART_DOUBLE_EXPERT: self.GAME_CHART_DOUBLE_EXPERT,
+            self.CHART_DOUBLE_CHALLENGE: self.GAME_CHART_DOUBLE_CHALLENGE,
+        }[db_chart]
+
+    def db_to_game_halo(self, db_halo: int) -> int:
+        if db_halo == self.HALO_MARVELOUS_FULL_COMBO:
+            combo_type = self.GAME_HALO_MARVELOUS_COMBO
+        elif db_halo == self.HALO_PERFECT_FULL_COMBO:
+            combo_type = self.GAME_HALO_PERFECT_COMBO
+        elif db_halo == self.HALO_GREAT_FULL_COMBO:
+            combo_type = self.GAME_HALO_GREAT_COMBO
+        elif db_halo == self.HALO_GOOD_FULL_COMBO:
+            combo_type = self.GAME_HALO_GOOD_COMBO
+        else:
+            combo_type = self.GAME_HALO_NONE
+        return combo_type
+
+    def handle_game_common_request(self, request: Node) -> Node:
+        game = Node.void('game')
+        for flagid in range(512):
+            flag = Node.void('flag')
+            game.add_child(flag)
+
+            flag.set_attribute('id', str(flagid))
+            flag.set_attribute('t', '0')
+            flag.set_attribute('s1', '0')
+            flag.set_attribute('s2', '0')
+            flag.set_attribute('area', '51')
+            flag.set_attribute('is_final', '0')
+
+        # Last month's hit chart
+        hit_chart = self.data.local.music.get_hit_chart(self.game, self.music_version, self.GAME_MAX_SONGS, 30)
+        counts_by_reflink = [0] * self.GAME_MAX_SONGS
+        for (reflink, plays) in hit_chart:
+            if reflink >= 0 and reflink < self.GAME_MAX_SONGS:
+                counts_by_reflink[reflink] = plays
+        game.add_child(Node.u32_array('cnt_music_monthly', counts_by_reflink))
+
+        # Last week's hit chart
+        hit_chart = self.data.local.music.get_hit_chart(self.game, self.music_version, self.GAME_MAX_SONGS, 7)
+        counts_by_reflink = [0] * self.GAME_MAX_SONGS
+        for (reflink, plays) in hit_chart:
+            if reflink >= 0 and reflink < self.GAME_MAX_SONGS:
+                counts_by_reflink[reflink] = plays
+        game.add_child(Node.u32_array('cnt_music_weekly', counts_by_reflink))
+
+        # Last day's hit chart
+        hit_chart = self.data.local.music.get_hit_chart(self.game, self.music_version, self.GAME_MAX_SONGS, 1)
+        counts_by_reflink = [0] * self.GAME_MAX_SONGS
+        for (reflink, plays) in hit_chart:
+            if reflink >= 0 and reflink < self.GAME_MAX_SONGS:
+                counts_by_reflink[reflink] = plays
+        game.add_child(Node.u32_array('cnt_music_daily', counts_by_reflink))
+
+        return game
+
+    def handle_game_load_m_request(self, request: Node) -> Node:
+        extid = intish(request.attribute('code'))
+        refid = request.attribute('refid')
+
+        if extid is not None:
+            # Rival score loading
+            userid = self.data.remote.user.from_extid(self.game, self.version, extid)
+        else:
+            # Self score loading
+            userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+
+        if userid is not None:
+            scores = self.data.remote.music.get_scores(self.game, self.music_version, userid)
+        else:
+            scores = []
+
+        sortedscores: Dict[int, Dict[int, Score]] = {}
+        for score in scores:
+            if score.id not in sortedscores:
+                sortedscores[score.id] = {}
+            sortedscores[score.id][score.chart] = score
+
+        game = Node.void('game')
+        for song in sortedscores:
+            music = Node.void('music')
+            game.add_child(music)
+            music.set_attribute('reclink', str(song))
+
+            for chart in sortedscores[song]:
+                score = sortedscores[song][chart]
+                try:
+                    gamechart = self.db_to_game_chart(chart)
+                except KeyError:
+                    # Don't support this chart in this game
+                    continue
+                gamerank = self.db_to_game_rank(score.data.get_int('rank'))
+                combo_type = self.db_to_game_halo(score.data.get_int('halo'))
+
+                typenode = Node.void('type')
+                music.add_child(typenode)
+                typenode.set_attribute('diff', str(gamechart))
+
+                typenode.add_child(Node.u32('score', score.points))
+                typenode.add_child(Node.u16('count', score.plays))
+                typenode.add_child(Node.u8('rank', gamerank))
+                typenode.add_child(Node.u8('combo_type', combo_type))
+                # The game optionally receives hard, life8, life4, risky, assist_clear, normal_clear
+                # u8 values too, and saves music scores with these set, but the UI doesn't appear to
+                # do anything with them, so we don't care.
+
+        return game
+
+    def handle_game_save_m_request(self, request: Node) -> Node:
+        refid = request.attribute('refid')
+        songid = int(request.attribute('mid'))
+        chart = self.game_to_db_chart(int(request.attribute('mtype')))
+
+        # Calculate statistics
+        data = request.child('data')
+        points = int(data.attribute('score'))
+        combo = int(data.attribute('combo'))
+        rank = self.game_to_db_rank(int(data.attribute('rank')))
+        if points == 1000000:
+            halo = self.HALO_MARVELOUS_FULL_COMBO
+        elif int(data.attribute('perf_fc')) != 0:
+            halo = self.HALO_PERFECT_FULL_COMBO
+        elif int(data.attribute('great_fc')) != 0:
+            halo = self.HALO_GREAT_FULL_COMBO
+        elif int(data.attribute('good_fc')) != 0:
+            halo = self.HALO_GOOD_FULL_COMBO
+        else:
+            halo = self.HALO_NONE
+        trace = request.child_value('trace')
+
+        # Save the score, regardless of whether we have a refid. If we save
+        # an anonymous score, it only goes into the DB to count against the
+        # number of plays for that song/chart.
+        userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        self.update_score(
+            userid,
+            songid,
+            chart,
+            points,
+            rank,
+            halo,
+            combo,
+            trace,
+        )
+
+        # No response needed
+        game = Node.void('game')
+        return game
+
+    def handle_game_load_edit_request(self, request: Node) -> Node:
+        return Node.void('game')
+
+    def handle_game_save_resultshot_request(self, request: Node) -> Node:
+        return Node.void('game')
+
+    def handle_game_trace_request(self, request: Node) -> Node:
+        # This is almost identical to 2013 and below, except it will never
+        # even try to request course traces, so we fork from common functionality.
+        extid = int(request.attribute('code'))
+        chart = int(request.attribute('type'))
+        mid = intish(request.attribute('mid'))
+
+        # Base packet is just game, if we find something we add to it
+        game = Node.void('game')
+
+        # Rival trace loading
+        userid = self.data.remote.user.from_extid(self.game, self.version, extid)
+        if userid is None:
+            # Nothing to load
+            return game
+
+        # Load trace from song score
+        songscore = self.data.remote.music.get_score(
+            self.game,
+            self.music_version,
+            userid,
+            mid,
+            self.game_to_db_chart(chart),
+        )
+        if songscore is not None and 'trace' in songscore.data:
+            game.add_child(Node.u32('size', len(songscore.data['trace'])))
+            game.add_child(Node.u8_array('trace', songscore.data['trace']))
+
+        return game
+
+    def format_profile(self, userid: UserID, profile: ValidatedDict) -> Node:
+        root = Node.void('game')
+
+        # Look up play stats we bridge to every mix
+        play_stats = self.get_play_statistics(userid)
+
+        # Basic game settings
+        root.add_child(Node.string('seq', ''))
+        root.add_child(Node.u32('code', profile.get_int('extid')))
+        root.add_child(Node.string('name', profile.get_str('name')))
+        root.add_child(Node.u8('area', profile.get_int('area', 51)))
+        root.add_child(Node.u32('cnt_s', play_stats.get_int('single_plays')))
+        root.add_child(Node.u32('cnt_d', play_stats.get_int('double_plays')))
+        root.add_child(Node.u32('cnt_b', play_stats.get_int('battle_plays')))  # This could be wrong, its a guess
+        root.add_child(Node.u32('cnt_m0', play_stats.get_int('cnt_m0')))
+        root.add_child(Node.u32('cnt_m1', play_stats.get_int('cnt_m1')))
+        root.add_child(Node.u32('cnt_m2', play_stats.get_int('cnt_m2')))
+        root.add_child(Node.u32('cnt_m3', play_stats.get_int('cnt_m3')))
+        root.add_child(Node.u32('cnt_m4', play_stats.get_int('cnt_m4')))
+        root.add_child(Node.u32('cnt_m5', play_stats.get_int('cnt_m5')))
+        root.add_child(Node.u32('exp', play_stats.get_int('exp')))
+        root.add_child(Node.u32('exp_o', profile.get_int('exp_o')))
+        root.add_child(Node.u32('star', profile.get_int('star')))
+        root.add_child(Node.u32('star_c', profile.get_int('star_c')))
+        root.add_child(Node.u8('combo', profile.get_int('combo', 0)))
+        root.add_child(Node.u8('timing_diff', profile.get_int('early_late', 0)))
+
+        # Character stuff
+        chara = Node.void('chara')
+        root.add_child(chara)
+        chara.set_attribute('my', str(profile.get_int('chara', 30)))
+        root.add_child(Node.u16_array('chara_opt', profile.get_int_array('chara_opt', 96, [208] * 96)))
+
+        # Drill rankings
+        if 'title_gr' in profile:
+            title_gr = Node.void('title_gr')
+            root.add_child(title_gr)
+            title_grdict = profile.get_dict('title_gr')
+            if 't' in title_grdict:
+                title_gr.set_attribute('t', str(title_grdict.get_int('t')))
+            if 's' in title_grdict:
+                title_gr.set_attribute('s', str(title_grdict.get_int('s')))
+            if 'd' in title_grdict:
+                title_gr.set_attribute('d', str(title_grdict.get_int('d')))
+
+        # Calorie mode
+        if 'weight' in profile:
+            workouts = self.data.local.user.get_time_based_achievements(
+                self.game,
+                self.version,
+                userid,
+                achievementtype='workout',
+                since=Time.now() - Time.SECONDS_IN_DAY,
+            )
+            total = sum([w.data.get_int('calories') for w in workouts])
+            workout = Node.void('workout')
+            root.add_child(workout)
+            workout.set_attribute('weight', str(profile.get_int('weight')))
+            workout.set_attribute('day', str(total))
+            workout.set_attribute('disp', '1')
+
+            # Unsure if this should be last day, or total calories ever
+            totalcalorie = Node.void('totalcalorie')
+            root.add_child(totalcalorie)
+            totalcalorie.set_attribute('total', str(total))
+
+        # Daily play counts
+        last_play_date = play_stats.get_int_array('last_play_date', 3)
+        today_play_date = Time.todays_date()
+        if (
+            last_play_date[0] == today_play_date[0] and
+            last_play_date[1] == today_play_date[1] and
+            last_play_date[2] == today_play_date[2]
+        ):
+            today_count = play_stats.get_int('today_plays', 0)
+        else:
+            today_count = 0
+        daycount = Node.void('daycount')
+        root.add_child(daycount)
+        daycount.set_attribute('playcount', str(today_count))
+
+        # Daily combo stuff, unknown how this works
+        dailycombo = Node.void('dailycombo')
+        root.add_child(dailycombo)
+        dailycombo.set_attribute('daily_combo', str(0))
+        dailycombo.set_attribute('daily_combo_lv', str(0))
+
+        # Last cursor settings
+        last = Node.void('last')
+        root.add_child(last)
+        lastdict = profile.get_dict('last')
+        last.set_attribute('rival1', str(lastdict.get_int('rival1', -1)))
+        last.set_attribute('rival2', str(lastdict.get_int('rival2', -1)))
+        last.set_attribute('rival3', str(lastdict.get_int('rival3', -1)))
+        last.set_attribute('fri', str(lastdict.get_int('rival1', -1)))  # This literally goes to the same memory in 2014
+        last.set_attribute('style', str(lastdict.get_int('style')))
+        last.set_attribute('mode', str(lastdict.get_int('mode')))
+        last.set_attribute('cate', str(lastdict.get_int('cate')))
+        last.set_attribute('sort', str(lastdict.get_int('sort')))
+        last.set_attribute('mid', str(lastdict.get_int('mid')))
+        last.set_attribute('mtype', str(lastdict.get_int('mtype')))
+        last.set_attribute('cid', str(lastdict.get_int('cid')))
+        last.set_attribute('ctype', str(lastdict.get_int('ctype')))
+        last.set_attribute('sid', str(lastdict.get_int('sid')))
+
+        # Result stars
+        result_star = Node.void('result_star')
+        root.add_child(result_star)
+        result_stars = profile.get_int_array('result_stars', 9)
+        for i in range(9):
+            result_star.set_attribute('slot{}'.format(i + 1), str(result_stars[i]))
+
+        # Groove gauge level-ups
+        gr_s = Node.void('gr_s')
+        root.add_child(gr_s)
+        index = 1
+        for entry in profile.get_int_array('gr_s', 5):
+            gr_s.set_attribute('gr{}'.format(index), str(entry))
+            index = index + 1
+
+        gr_d = Node.void('gr_d')
+        root.add_child(gr_d)
+        index = 1
+        for entry in profile.get_int_array('gr_d', 5):
+            gr_d.set_attribute('gr{}'.format(index), str(entry))
+            index = index + 1
+
+        # Options in menus
+        root.add_child(Node.s16_array('opt', profile.get_int_array('opt', 16)))
+        root.add_child(Node.s16_array('opt_ex', profile.get_int_array('opt_ex', 16)))
+        option_ver = Node.void('option_ver')
+        root.add_child(option_ver)
+        option_ver.set_attribute('ver', str(profile.get_int('option_ver', 2)))
+        if 'option_02' in profile:
+            root.add_child(Node.s16_array('option_02', profile.get_int_array('option_02', 24)))
+
+        # Unlock flags
+        root.add_child(Node.u8_array('flag', profile.get_int_array('flag', 512, [1] * 512)[:256]))
+        root.add_child(Node.u8_array('flag_ex', profile.get_int_array('flag', 512, [1] * 512)))
+
+        # Ranking display?
+        root.add_child(Node.u16_array('rank', profile.get_int_array('rank', 100)))
+
+        # Rivals
+        links = self.data.local.user.get_links(self.game, self.version, userid)
+        for link in links:
+            if link.type[:7] != 'friend_':
+                continue
+
+            pos = int(link.type[7:])
+            friend = self.get_profile(link.other_userid)
+            play_stats = self.get_play_statistics(link.other_userid)
+            if friend is not None:
+                friendnode = Node.void('friend')
+                root.add_child(friendnode)
+                friendnode.set_attribute('pos', str(pos))
+                friendnode.set_attribute('vs', '0')
+                friendnode.set_attribute('up', '0')
+                friendnode.add_child(Node.u32('code', friend.get_int('extid')))
+                friendnode.add_child(Node.string('name', friend.get_str('name')))
+                friendnode.add_child(Node.u8('area', friend.get_int('area', 51)))
+                friendnode.add_child(Node.u32('exp', play_stats.get_int('exp')))
+                friendnode.add_child(Node.u32('star', friend.get_int('star')))
+
+                # Drill rankings
+                if 'title' in friend:
+                    title = Node.void('title')
+                    friendnode.add_child(title)
+                    titledict = friend.get_dict('title')
+                    if 't' in titledict:
+                        title.set_attribute('t', str(titledict.get_int('t')))
+                    if 's' in titledict:
+                        title.set_attribute('s', str(titledict.get_int('s')))
+                    if 'd' in titledict:
+                        title.set_attribute('d', str(titledict.get_int('d')))
+
+                if 'title_gr' in friend:
+                    title_gr = Node.void('title_gr')
+                    friendnode.add_child(title_gr)
+                    title_grdict = friend.get_dict('title_gr')
+                    if 't' in title_grdict:
+                        title_gr.set_attribute('t', str(title_grdict.get_int('t')))
+                    if 's' in title_grdict:
+                        title_gr.set_attribute('s', str(title_grdict.get_int('s')))
+                    if 'd' in title_grdict:
+                        title_gr.set_attribute('d', str(title_grdict.get_int('d')))
+
+                # Groove gauge level-ups
+                gr_s = Node.void('gr_s')
+                friendnode.add_child(gr_s)
+                index = 1
+                for entry in friend.get_int_array('gr_s', 5):
+                    gr_s.set_attribute('gr{}'.format(index), str(entry))
+                    index = index + 1
+
+                gr_d = Node.void('gr_d')
+                friendnode.add_child(gr_d)
+                index = 1
+                for entry in friend.get_int_array('gr_d', 5):
+                    gr_d.set_attribute('gr{}'.format(index), str(entry))
+                    index = index + 1
+
+        # Target stuff
+        target = Node.void('target')
+        root.add_child(target)
+        target.set_attribute('flag', str(profile.get_int('target_flag')))
+        target.set_attribute('setnum', str(profile.get_int('target_setnum')))
+
+        # Play area
+        areas = profile.get_int_array('play_area', 55)
+        play_area = Node.void('play_area')
+        root.add_child(play_area)
+        for i in range(len(areas)):
+            play_area.set_attribute('play_cnt{}'.format(i), str(areas[i]))
+
+        return root
+
+    def unformat_profile(self, userid: UserID, request: Node, oldprofile: ValidatedDict) -> ValidatedDict:
+        newprofile = copy.deepcopy(oldprofile)
+        play_stats = self.get_play_statistics(userid)
+
+        # Grab last node and accessories so we can make decisions based on type
+        last = request.child('last')
+        lastdict = newprofile.get_dict('last')
+        mode = int(last.attribute('mode'))
+        style = int(last.attribute('style'))
+        is_dp = style == self.GAME_STYLE_DOUBLE
+
+        # Drill rankings
+        title = request.child('title')
+        title_gr = request.child('title_gr')
+        titledict = newprofile.get_dict('title')
+        title_grdict = newprofile.get_dict('title_gr')
+
+        # Groove radar level ups
+        gr = request.child('gr')
+
+        # Set the correct values depending on if we're single or double play
+        if is_dp:
+            play_stats.increment_int('double_plays')
+            if gr is not None:
+                newprofile.replace_int_array(
+                    'gr_d',
+                    5,
+                    [
+                        intish(gr.attribute('gr1')),
+                        intish(gr.attribute('gr2')),
+                        intish(gr.attribute('gr3')),
+                        intish(gr.attribute('gr4')),
+                        intish(gr.attribute('gr5')),
+                    ],
+                )
+            if title is not None:
+                titledict.replace_int('d', title.value)
+                newprofile.replace_dict('title', titledict)
+            if title_gr is not None:
+                title_grdict.replace_int('d', title.value)
+                newprofile.replace_dict('title_gr', title_grdict)
+        else:
+            play_stats.increment_int('single_plays')
+            if gr is not None:
+                newprofile.replace_int_array(
+                    'gr_s',
+                    5,
+                    [
+                        intish(gr.attribute('gr1')),
+                        intish(gr.attribute('gr2')),
+                        intish(gr.attribute('gr3')),
+                        intish(gr.attribute('gr4')),
+                        intish(gr.attribute('gr5')),
+                    ],
+                )
+            if title is not None:
+                titledict.replace_int('s', title.value)
+                newprofile.replace_dict('title', titledict)
+            if title_gr is not None:
+                title_grdict.replace_int('s', title.value)
+                newprofile.replace_dict('title_gr', title_grdict)
+        play_stats.increment_int('cnt_m{}'.format(mode))
+
+        # Result stars
+        result_star = request.child('result_star')
+        if result_star is not None:
+            newprofile.replace_int_array(
+                'result_stars',
+                9,
+                [
+                    intish(result_star.attribute('slot1')),
+                    intish(result_star.attribute('slot2')),
+                    intish(result_star.attribute('slot3')),
+                    intish(result_star.attribute('slot4')),
+                    intish(result_star.attribute('slot5')),
+                    intish(result_star.attribute('slot6')),
+                    intish(result_star.attribute('slot7')),
+                    intish(result_star.attribute('slot8')),
+                    intish(result_star.attribute('slot9')),
+                ],
+            )
+
+        # Target stuff
+        target = request.child('target')
+        if target is not None:
+            newprofile.replace_int('target_flag', intish(target.attribute('flag')))
+            newprofile.replace_int('target_setnum', intish(target.attribute('setnum')))
+
+        # Update last attributes
+        lastdict.replace_int('rival1', intish(last.attribute('rival1')))
+        lastdict.replace_int('rival2', intish(last.attribute('rival2')))
+        lastdict.replace_int('rival3', intish(last.attribute('rival3')))
+        lastdict.replace_int('style', intish(last.attribute('style')))
+        lastdict.replace_int('mode', intish(last.attribute('mode')))
+        lastdict.replace_int('cate', intish(last.attribute('cate')))
+        lastdict.replace_int('sort', intish(last.attribute('sort')))
+        lastdict.replace_int('mid', intish(last.attribute('mid')))
+        lastdict.replace_int('mtype', intish(last.attribute('mtype')))
+        lastdict.replace_int('cid', intish(last.attribute('cid')))
+        lastdict.replace_int('ctype', intish(last.attribute('ctype')))
+        lastdict.replace_int('sid', intish(last.attribute('sid')))
+        newprofile.replace_dict('last', lastdict)
+
+        # Grab character options
+        chara = request.child('chara')
+        if chara is not None:
+            newprofile.replace_int('chara', intish(chara.attribute('my')))
+        chara_opt = request.child('chara_opt')
+        if chara_opt is not None:
+            # A bug in old versions of AVS returns the wrong number for set
+            newprofile.replace_int_array('chara_opt', 96, chara_opt.value[:96])
+
+        # Options
+        option_02 = request.child('option_02')
+        if option_02 is not None:
+            # A bug in old versions of AVS returns the wrong number for set
+            newprofile.replace_int_array('option_02', 24, option_02.value[:24])
+
+        # Experience and stars
+        exp = request.child_value('exp')
+        if exp is not None:
+            play_stats.replace_int('exp', play_stats.get_int('exp') + exp)
+        star = request.child_value('star')
+        if star is not None:
+            newprofile.replace_int('star', newprofile.get_int('star') + star)
+        star_c = request.child_value('star_c')
+        if star_c is not None:
+            newprofile.replace_int('star_c', newprofile.get_int('star_c') + exp)
+
+        # Update game flags
+        for child in request.children:
+            if child.name not in ['flag', 'flag_ex']:
+                continue
+            try:
+                value = int(child.attribute('data'))
+                offset = int(child.attribute('no'))
+            except ValueError:
+                continue
+
+            flags = newprofile.get_int_array('flag', 512, [1] * 512)
+            if offset < 0 or offset >= len(flags):
+                continue
+            flags[offset] = value
+            newprofile.replace_int_array('flag', 512, flags)
+
+        # Workout mode support
+        newweight = -1
+        oldweight = newprofile.get_int('weight')
+        for child in request.children:
+            if child.name != 'weight':
+                continue
+            newweight = child.value
+        if newweight < 0:
+            newweight = oldweight
+
+        # Either update or unset the weight depending on the game
+        if newweight == 0:
+            # Weight is unset or we declined to use this feature, remove from profile
+            if 'weight' in newprofile:
+                del newprofile['weight']
+        else:
+            # Weight has been set or previously retrieved, we should save calories
+            newprofile.replace_int('weight', newweight)
+            total = 0
+            for child in request.children:
+                if child.name != 'calory':
+                    continue
+                total += child.value
+            self.data.local.user.put_time_based_achievement(
+                self.game,
+                self.version,
+                userid,
+                0,
+                'workout',
+                {
+                    'calories': total,
+                    'weight': newweight,
+                },
+            )
+
+        # Look up old friends
+        oldfriends: List[Optional[UserID]] = [None] * 10
+        links = self.data.local.user.get_links(self.game, self.version, userid)
+        for link in links:
+            if link.type[:7] != 'friend_':
+                continue
+
+            pos = int(link.type[7:])
+            oldfriends[pos] = link.other_userid
+
+        # Save any rivals that were added/removed/changed
+        newfriends = oldfriends[:]
+        for child in request.children:
+            if child.name != 'friend':
+                continue
+
+            code = int(child.attribute('code'))
+            pos = int(child.attribute('pos'))
+
+            if pos >= 0 and pos < 10:
+                if code == 0:
+                    # We cleared this friend
+                    newfriends[pos] = None
+                else:
+                    # Try looking up the userid
+                    newfriends[pos] = self.data.remote.user.from_extid(self.game, self.version, code)
+
+        # Diff the set of links to determine updates
+        for i in range(10):
+            if newfriends[i] == oldfriends[i]:
+                continue
+
+            if newfriends[i] is None:
+                # Kill the rival in this location
+                self.data.local.user.destroy_link(
+                    self.game,
+                    self.version,
+                    userid,
+                    'friend_{}'.format(i),
+                    oldfriends[i],
+                )
+            elif oldfriends[i] is None:
+                # Add rival in this location
+                self.data.local.user.put_link(
+                    self.game,
+                    self.version,
+                    userid,
+                    'friend_{}'.format(i),
+                    newfriends[i],
+                    {},
+                )
+            else:
+                # Changed the rival here, kill the old one, add the new one
+                self.data.local.user.destroy_link(
+                    self.game,
+                    self.version,
+                    userid,
+                    'friend_{}'.format(i),
+                    oldfriends[i],
+                )
+                self.data.local.user.put_link(
+                    self.game,
+                    self.version,
+                    userid,
+                    'friend_{}'.format(i),
+                    newfriends[i],
+                    {},
+                )
+
+        # Play area counter
+        shop_area = int(request.attribute('shop_area'))
+        if shop_area >= 0 and shop_area < 55:
+            areas = newprofile.get_int_array('play_area', 55)
+            areas[shop_area] = areas[shop_area] + 1
+            newprofile.replace_int_array('play_area', 55, areas)
+
+        # Keep track of play statistics
+        self.update_play_statistics(userid, play_stats)
+
+        return newprofile

bemani/backend/ddr/ddra20.py

@@ -0,0 +1,26 @@
+# vim: set fileencoding=utf-8
+from typing import Optional
+
+from bemani.backend.ddr.base import DDRBase
+from bemani.backend.ddr.ddrace import DDRAce
+from bemani.common import VersionConstants
+
+
+class DDRA20(
+    DDRBase,
+):
+
+    name = 'DanceDanceRevolution A20'
+    version = VersionConstants.DDR_A20
+
+    def previous_version(self) -> Optional[DDRBase]:
+        return DDRAce(self.data, self.config, self.model)
+
+    def supports_paseli(self) -> bool:
+        if self.model.dest != 'J':
+            # DDR Ace in USA mode doesn't support PASELI properly.
+            # When in Asia mode it shows PASELI but won't let you select it.
+            return False
+        else:
+            # All other modes should work with PASELI.
+            return True

bemani/backend/ddr/ddrace.py

@@ -0,0 +1,801 @@
+# vim: set fileencoding=utf-8
+import base64
+from typing import Dict, List, Optional
+
+from bemani.backend.ess import EventLogHandler
+from bemani.backend.ddr.base import DDRBase
+from bemani.backend.ddr.ddr2014 import DDR2014
+from bemani.common import ValidatedDict, VersionConstants, CardCipher, Time, ID, intish
+from bemani.data import Achievement, Machine, Score, UserID
+from bemani.protocol import Node
+
+
+class DDRAce(
+    DDRBase,
+    EventLogHandler,
+):
+
+    name = 'DanceDanceRevolution A'
+    version = VersionConstants.DDR_ACE
+
+    GAME_STYLE_SINGLE = 0
+    GAME_STYLE_DOUBLE = 1
+    GAME_STYLE_VERSUS = 2
+
+    GAME_RIVAL_TYPE_RIVAL3 = 32
+    GAME_RIVAL_TYPE_RIVAL2 = 16
+    GAME_RIVAL_TYPE_RIVAL1 = 8
+    GAME_RIVAL_TYPE_WORLD = 4
+    GAME_RIVAL_TYPE_AREA = 2
+    GAME_RIVAL_TYPE_MACHINE = 1
+
+    GAME_CHART_SINGLE_BEGINNER = 0
+    GAME_CHART_SINGLE_BASIC = 1
+    GAME_CHART_SINGLE_DIFFICULT = 2
+    GAME_CHART_SINGLE_EXPERT = 3
+    GAME_CHART_SINGLE_CHALLENGE = 4
+    GAME_CHART_DOUBLE_BASIC = 5
+    GAME_CHART_DOUBLE_DIFFICULT = 6
+    GAME_CHART_DOUBLE_EXPERT = 7
+    GAME_CHART_DOUBLE_CHALLENGE = 8
+
+    GAME_HALO_NONE = 6
+    GAME_HALO_GOOD_COMBO = 7
+    GAME_HALO_GREAT_COMBO = 8
+    GAME_HALO_PERFECT_COMBO = 9
+    GAME_HALO_MARVELOUS_COMBO = 10
+
+    GAME_RANK_E = 15
+    GAME_RANK_D = 14
+    GAME_RANK_D_PLUS = 13
+    GAME_RANK_C_MINUS = 12
+    GAME_RANK_C = 11
+    GAME_RANK_C_PLUS = 10
+    GAME_RANK_B_MINUS = 9
+    GAME_RANK_B = 8
+    GAME_RANK_B_PLUS = 7
+    GAME_RANK_A_MINUS = 6
+    GAME_RANK_A = 5
+    GAME_RANK_A_PLUS = 4
+    GAME_RANK_AA_MINUS = 3
+    GAME_RANK_AA = 2
+    GAME_RANK_AA_PLUS = 1
+    GAME_RANK_AAA = 0
+
+    GAME_MAX_SONGS = 1024
+
+    GAME_COMMON_AREA_OFFSET = 1
+    GAME_COMMON_WEIGHT_DISPLAY_OFFSET = 3
+    GAME_COMMON_CHARACTER_OFFSET = 4
+    GAME_COMMON_EXTRA_CHARGE_OFFSET = 5
+    GAME_COMMON_TOTAL_PLAYS_OFFSET = 9
+    GAME_COMMON_SINGLE_PLAYS_OFFSET = 11
+    GAME_COMMON_DOUBLE_PLAYS_OFFSET = 12
+    GAME_COMMON_WEIGHT_OFFSET = 17
+    GAME_COMMON_NAME_OFFSET = 25
+    GAME_COMMON_SEQ_OFFSET = 26
+
+    GAME_OPTION_SCROLL_OFFSET = 1
+    GAME_OPTION_BOOST_OFFSET = 2
+    GAME_OPTION_APPEARANCE_OFFSET = 3
+    GAME_OPTION_TURN_OFFSET = 4
+    GAME_OPTION_STEP_ZONE_OFFSET = 5
+    GAME_OPTION_SCROLL_OFFSET = 6
+    GAME_OPTION_ARROW_COLOR_OFFSET = 7
+    GAME_OPTION_CUT_OFFSET = 8
+    GAME_OPTION_FREEZE_OFFSET = 9
+    GAME_OPTION_JUMPS_OFFSET = 10
+    GAME_OPTION_ARROW_SKIN_OFFSET = 11
+    GAME_OPTION_FILTER_OFFSET = 12
+    GAME_OPTION_GUIDELINE_OFFSET = 13
+    GAME_OPTION_GAUGE_OFFSET = 14
+    GAME_OPTION_COMBO_POSITION_OFFSET = 15
+    GAME_OPTION_FAST_SLOW_OFFSET = 16
+
+    GAME_LAST_CALORIES_OFFSET = 10
+
+    GAME_RIVAL_SLOT_1_ACTIVE_OFFSET = 1
+    GAME_RIVAL_SLOT_2_ACTIVE_OFFSET = 2
+    GAME_RIVAL_SLOT_3_ACTIVE_OFFSET = 3
+    GAME_RIVAL_SLOT_1_DDRCODE_OFFSET = 9
+    GAME_RIVAL_SLOT_2_DDRCODE_OFFSET = 10
+    GAME_RIVAL_SLOT_3_DDRCODE_OFFSET = 11
+
+    def previous_version(self) -> Optional[DDRBase]:
+        return DDR2014(self.data, self.config, self.model)
+
+    def supports_paseli(self) -> bool:
+        if self.model.dest != 'J':
+            # DDR Ace in USA mode doesn't support PASELI properly.
+            # When in Asia mode it shows PASELI but won't let you select it.
+            return False
+        else:
+            # All other modes should work with PASELI.
+            return True
+
+    def game_to_db_rank(self, game_rank: int) -> int:
+        return {
+            self.GAME_RANK_AAA: self.RANK_AAA,
+            self.GAME_RANK_AA_PLUS: self.RANK_AA_PLUS,
+            self.GAME_RANK_AA: self.RANK_AA,
+            self.GAME_RANK_AA_MINUS: self.RANK_AA_MINUS,
+            self.GAME_RANK_A_PLUS: self.RANK_A_PLUS,
+            self.GAME_RANK_A: self.RANK_A,
+            self.GAME_RANK_A_MINUS: self.RANK_A_MINUS,
+            self.GAME_RANK_B_PLUS: self.RANK_B_PLUS,
+            self.GAME_RANK_B: self.RANK_B,
+            self.GAME_RANK_B_MINUS: self.RANK_B_MINUS,
+            self.GAME_RANK_C_PLUS: self.RANK_C_PLUS,
+            self.GAME_RANK_C: self.RANK_C,
+            self.GAME_RANK_C_MINUS: self.RANK_C_MINUS,
+            self.GAME_RANK_D_PLUS: self.RANK_D_PLUS,
+            self.GAME_RANK_D: self.RANK_D,
+            self.GAME_RANK_E: self.RANK_E,
+        }[game_rank]
+
+    def db_to_game_rank(self, db_rank: int) -> int:
+        return {
+            self.RANK_AAA: self.GAME_RANK_AAA,
+            self.RANK_AA_PLUS: self.GAME_RANK_AA_PLUS,
+            self.RANK_AA: self.GAME_RANK_AA,
+            self.RANK_AA_MINUS: self.GAME_RANK_AA_MINUS,
+            self.RANK_A_PLUS: self.GAME_RANK_A_PLUS,
+            self.RANK_A: self.GAME_RANK_A,
+            self.RANK_A_MINUS: self.GAME_RANK_A_MINUS,
+            self.RANK_B_PLUS: self.GAME_RANK_B_PLUS,
+            self.RANK_B: self.GAME_RANK_B,
+            self.RANK_B_MINUS: self.GAME_RANK_B_MINUS,
+            self.RANK_C_PLUS: self.GAME_RANK_C_PLUS,
+            self.RANK_C: self.GAME_RANK_C,
+            self.RANK_C_MINUS: self.GAME_RANK_C_MINUS,
+            self.RANK_D_PLUS: self.GAME_RANK_D_PLUS,
+            self.RANK_D: self.GAME_RANK_D,
+            self.RANK_E: self.GAME_RANK_E,
+        }[db_rank]
+
+    def game_to_db_chart(self, game_chart: int) -> int:
+        return {
+            self.GAME_CHART_SINGLE_BEGINNER: self.CHART_SINGLE_BEGINNER,
+            self.GAME_CHART_SINGLE_BASIC: self.CHART_SINGLE_BASIC,
+            self.GAME_CHART_SINGLE_DIFFICULT: self.CHART_SINGLE_DIFFICULT,
+            self.GAME_CHART_SINGLE_EXPERT: self.CHART_SINGLE_EXPERT,
+            self.GAME_CHART_SINGLE_CHALLENGE: self.CHART_SINGLE_CHALLENGE,
+            self.GAME_CHART_DOUBLE_BASIC: self.CHART_DOUBLE_BASIC,
+            self.GAME_CHART_DOUBLE_DIFFICULT: self.CHART_DOUBLE_DIFFICULT,
+            self.GAME_CHART_DOUBLE_EXPERT: self.CHART_DOUBLE_EXPERT,
+            self.GAME_CHART_DOUBLE_CHALLENGE: self.CHART_DOUBLE_CHALLENGE,
+        }[game_chart]
+
+    def db_to_game_chart(self, db_chart: int) -> int:
+        return {
+            self.CHART_SINGLE_BEGINNER: self.GAME_CHART_SINGLE_BEGINNER,
+            self.CHART_SINGLE_BASIC: self.GAME_CHART_SINGLE_BASIC,
+            self.CHART_SINGLE_DIFFICULT: self.GAME_CHART_SINGLE_DIFFICULT,
+            self.CHART_SINGLE_EXPERT: self.GAME_CHART_SINGLE_EXPERT,
+            self.CHART_SINGLE_CHALLENGE: self.GAME_CHART_SINGLE_CHALLENGE,
+            self.CHART_DOUBLE_BASIC: self.GAME_CHART_DOUBLE_BASIC,
+            self.CHART_DOUBLE_DIFFICULT: self.GAME_CHART_DOUBLE_DIFFICULT,
+            self.CHART_DOUBLE_EXPERT: self.GAME_CHART_DOUBLE_EXPERT,
+            self.CHART_DOUBLE_CHALLENGE: self.GAME_CHART_DOUBLE_CHALLENGE,
+        }[db_chart]
+
+    def game_to_db_halo(self, game_halo: int) -> int:
+        if game_halo == self.GAME_HALO_MARVELOUS_COMBO:
+            return self.HALO_MARVELOUS_FULL_COMBO
+        elif game_halo == self.GAME_HALO_PERFECT_COMBO:
+            return self.HALO_PERFECT_FULL_COMBO
+        elif game_halo == self.GAME_HALO_GREAT_COMBO:
+            return self.HALO_GREAT_FULL_COMBO
+        elif game_halo == self.GAME_HALO_GOOD_COMBO:
+            return self.HALO_GOOD_FULL_COMBO
+        else:
+            return self.HALO_NONE
+
+    def db_to_game_halo(self, db_halo: int) -> int:
+        if db_halo == self.HALO_MARVELOUS_FULL_COMBO:
+            return self.GAME_HALO_MARVELOUS_COMBO
+        elif db_halo == self.HALO_PERFECT_FULL_COMBO:
+            return self.GAME_HALO_PERFECT_COMBO
+        elif db_halo == self.HALO_GREAT_FULL_COMBO:
+            return self.GAME_HALO_GREAT_COMBO
+        elif db_halo == self.HALO_GOOD_FULL_COMBO:
+            return self.GAME_HALO_GOOD_COMBO
+        else:
+            return self.GAME_HALO_NONE
+
+    def handle_tax_get_phase_request(self, request: Node) -> Node:
+        tax = Node.void('tax')
+        tax.add_child(Node.s32('phase', 0))
+        return tax
+
+    def __handle_userload(self, userid: Optional[UserID], requestdata: Node, response: Node) -> None:
+        has_profile: bool = False
+        achievements: List[Achievement] = []
+        scores: List[Score] = []
+
+        if userid is not None:
+            has_profile = self.has_profile(userid)
+            achievements = self.data.local.user.get_achievements(self.game, self.version, userid)
+            scores = self.data.remote.music.get_scores(self.game, self.music_version, userid)
+
+        # Place scores into an arrangement for easier distribution to Ace.
+        scores_by_mcode: Dict[int, List[Optional[Score]]] = {}
+        for score in scores:
+            if score.id not in scores_by_mcode:
+                scores_by_mcode[score.id] = [None] * 9
+
+            scores_by_mcode[score.id][self.db_to_game_chart(score.chart)] = score
+
+        # First, set new flag
+        response.add_child(Node.bool('is_new', not has_profile))
+
+        # Now, return the scores to Ace
+        for mcode in scores_by_mcode:
+            music = Node.void('music')
+            response.add_child(music)
+            music.add_child(Node.u32('mcode', mcode))
+
+            scores_that_matter = scores_by_mcode[mcode]
+            while scores_that_matter[-1] is None:
+                scores_that_matter = scores_that_matter[:-1]
+
+            for score in scores_that_matter:
+                note = Node.void('note')
+                music.add_child(note)
+
+                if score is None:
+                    note.add_child(Node.u16('count', 0))
+                    note.add_child(Node.u8('rank', 0))
+                    note.add_child(Node.u8('clearkind', 0))
+                    note.add_child(Node.s32('score', 0))
+                    note.add_child(Node.s32('ghostid', 0))
+                else:
+                    note.add_child(Node.u16('count', score.plays))
+                    note.add_child(Node.u8('rank', self.db_to_game_rank(score.data.get_int('rank'))))
+                    note.add_child(Node.u8('clearkind', self.db_to_game_halo(score.data.get_int('halo'))))
+                    note.add_child(Node.s32('score', score.points))
+                    note.add_child(Node.s32('ghostid', score.key))
+
+        # Active event settings
+        activeevents = [
+            1,
+            3,
+            5,
+            9,
+            10,
+            11,
+            12,
+            13,
+            15,
+            16,
+            17,
+            18,
+            19,
+            20,
+            21,
+            22,
+            23,
+            24,
+            25,
+            26,
+            27,
+            28,
+            29,
+            30,
+            31,
+            32,
+            33,
+            34,
+            35,
+            36,
+            37,
+            38,
+            39,
+            40,
+            41,
+            42,
+        ]
+
+        # Event reward settings
+        rewards = {
+            '30': {
+                999: 5,
+            }
+        }
+
+        # Now handle event progress and activation.
+        events = {ach.id: ach.data for ach in achievements if ach.type == '9999'}
+        progress = [ach for ach in achievements if ach.type != '9999']
+
+        # Make sure we always send a babylon's adventure save event or the game won't send progress
+        babylon_included = False
+        for evtprogress in progress:
+            if evtprogress.id == 999 and evtprogress.type == '30':
+                babylon_included = True
+                break
+
+        if not babylon_included:
+            progress.append(Achievement(
+                999,
+                '30',
+                None,
+                {
+                    'completed': False,
+                    'progress': 0,
+                },
+            ))
+
+        for event in activeevents:
+            # Get completion data
+            playerstats = events.get(event, ValidatedDict({'completed': False}))
+
+            # Return the data
+            eventdata = Node.void('eventdata')
+            response.add_child(eventdata)
+            eventdata.add_child(Node.u32('eventid', event))
+            eventdata.add_child(Node.s32('eventtype', 9999))
+            eventdata.add_child(Node.u32('eventno', 0))
+            eventdata.add_child(Node.s64('condition', 0))
+            eventdata.add_child(Node.u32('reward', 0))
+            eventdata.add_child(Node.s32('comptime', 1 if playerstats.get_bool('completed') else 0))
+            eventdata.add_child(Node.s64('savedata', 0))
+
+        for evtprogress in progress:
+            # Babylon's adventure progres and anything else the game sends
+            eventdata = Node.void('eventdata')
+            response.add_child(eventdata)
+            eventdata.add_child(Node.u32('eventid', evtprogress.id))
+            eventdata.add_child(Node.s32('eventtype', int(evtprogress.type)))
+            eventdata.add_child(Node.u32('eventno', 0))
+            eventdata.add_child(Node.s64('condition', 0))
+            eventdata.add_child(Node.u32('reward', rewards.get(evtprogress.type, {}).get(evtprogress.id)))
+            eventdata.add_child(Node.s32('comptime', 1 if evtprogress.data.get_bool('completed') else 0))
+            eventdata.add_child(Node.s64('savedata', evtprogress.data.get_int('progress')))
+
+    def __handle_usersave(self, userid: Optional[UserID], requestdata: Node, response: Node) -> None:
+        if userid is None:
+            # the game sends us empty user ID strings when a guest is playing.
+            # Return early so it doesn't wait a minute and a half to show the
+            # results screen.
+            return
+
+        if requestdata.child_value('isgameover'):
+            style = int(requestdata.child_value('playstyle'))
+            is_dp = style == self.GAME_STYLE_DOUBLE
+
+            # We don't save anything for gameover requests, since we
+            # already saved scores on individual ones. So, just use this
+            # as a spot to bump play counts and such
+            play_stats = self.get_play_statistics(userid)
+            if is_dp:
+                play_stats.increment_int('double_plays')
+            else:
+                play_stats.increment_int('single_plays')
+            self.update_play_statistics(userid, play_stats)
+
+            # Now is a good time to check if we have workout mode enabled,
+            # and if so, store the calories earned for this set.
+            profile = self.get_profile(userid)
+            enabled = profile.get_bool('workout_mode')
+            weight = profile.get_int('weight')
+
+            if enabled and weight > 0:
+                # We enabled weight display, find the calories and save them
+                total = 0
+                for child in requestdata.children:
+                    if child.name != 'note':
+                        continue
+
+                    total = total + (child.child_value('calorie') or 0)
+
+                self.data.local.user.put_time_based_achievement(
+                    self.game,
+                    self.version,
+                    userid,
+                    0,
+                    'workout',
+                    {
+                        'calories': total,
+                        'weight': weight,
+                    },
+                )
+
+            # Find any event updates
+            for child in requestdata.children:
+                if child.name != 'event':
+                    continue
+
+                # Skip empty events or events we don't support
+                eventid = child.child_value('eventid')
+                eventtype = child.child_value('eventtype')
+                if eventid == 0 or eventtype == 0:
+                    continue
+
+                # Save data to replay to the client later
+                completed = child.child_value('comptime') != 0
+                progress = child.child_value('savedata')
+
+                self.data.local.user.put_achievement(
+                    self.game,
+                    self.version,
+                    userid,
+                    eventid,
+                    str(eventtype),
+                    {
+                        'completed': completed,
+                        'progress': progress,
+                    },
+                )
+
+            return
+
+        # Find the highest stagenum played
+        score = None
+        stagenum = 0
+        for child in requestdata.children:
+            if child.name != 'note':
+                continue
+
+            if child.child_value('stagenum') > stagenum:
+                score = child
+                stagenum = child.child_value('stagenum')
+
+        if score is None:
+            raise Exception('Couldn\'t find newest score to save!')
+
+        songid = score.child_value('mcode')
+        chart = self.game_to_db_chart(score.child_value('notetype'))
+        rank = self.game_to_db_rank(score.child_value('rank'))
+        halo = self.game_to_db_halo(score.child_value('clearkind'))
+        points = score.child_value('score')
+        combo = score.child_value('maxcombo')
+        ghost = score.child_value('ghost')
+        self.update_score(
+            userid,
+            songid,
+            chart,
+            points,
+            rank,
+            halo,
+            combo,
+            ghost=ghost,
+        )
+
+    def __handle_rivalload(self, userid: Optional[UserID], requestdata: Node, response: Node) -> None:
+        data = Node.void('data')
+        response.add_child(data)
+        data.add_child(Node.s32('recordtype', requestdata.child_value('loadflag')))
+
+        thismachine = self.data.local.machine.get_machine(self.config['machine']['pcbid'])
+        machines_by_id: Dict[int, Optional[Machine]] = {thismachine.id: thismachine}
+
+        loadkind = requestdata.child_value('loadflag')
+        profiles_by_userid: Dict[UserID, ValidatedDict] = {}
+
+        def get_machine(lid: int) -> Optional[Machine]:
+            if lid not in machines_by_id:
+                pcbid = self.data.local.machine.from_machine_id(lid)
+                if pcbid is None:
+                    machines_by_id[lid] = None
+                    return None
+
+                machine = self.data.local.machine.get_machine(pcbid)
+                if machine is None:
+                    machines_by_id[lid] = None
+                    return None
+
+                machines_by_id[lid] = machine
+            return machines_by_id[lid]
+
+        if loadkind == self.GAME_RIVAL_TYPE_WORLD:
+            # Just load all scores for this network
+            scores = self.data.remote.music.get_all_records(self.game, self.music_version)
+        elif loadkind == self.GAME_RIVAL_TYPE_AREA:
+            if thismachine.arcade is not None:
+                match_arcade = thismachine.arcade
+                match_machine = None
+            else:
+                match_arcade = None
+                match_machine = thismachine.id
+
+            # Load up all scores by any user registered on a machine in the same arcade
+            profiles = self.data.local.user.get_all_profiles(self.game, self.version)
+            userids: List[UserID] = []
+            for userid, profiledata in profiles:
+                profiles_by_userid[userid] = profiledata
+
+                # If we have an arcade to match, see if this user's location matches the arcade.
+                # If we don't, just match lid directly
+                if match_arcade is not None:
+                    theirmachine = get_machine(profiledata.get_int('lid'))
+                    if theirmachine is not None and theirmachine.arcade == match_arcade:
+                        userids.append(userid)
+                elif match_machine is not None:
+                    if profiledata.get_int('lid') == match_machine:
+                        userids.append(userid)
+
+            # Load all scores for users in the area
+            scores = self.data.local.music.get_all_records(self.game, self.music_version, userlist=userids)
+        elif loadkind == self.GAME_RIVAL_TYPE_MACHINE:
+            # Load up all scores and filter them by those earned at this location
+            scores = self.data.local.music.get_all_records(self.game, self.music_version, locationlist=[thismachine.id])
+        elif loadkind in [
+            self.GAME_RIVAL_TYPE_RIVAL1,
+            self.GAME_RIVAL_TYPE_RIVAL2,
+            self.GAME_RIVAL_TYPE_RIVAL3,
+        ]:
+            # Load up this user's highscores, format the way the below code expects it
+            extid = requestdata.child_value('ddrcode')
+            otherid = self.data.remote.user.from_extid(self.game, self.version, extid)
+            userscores = self.data.remote.music.get_scores(self.game, self.music_version, otherid)
+            scores = [(otherid, score) for score in userscores]
+        else:
+            # Nothing here
+            scores = []
+
+        missing_users = [userid for (userid, _) in scores if userid not in profiles_by_userid]
+        for (userid, profile) in self.get_any_profiles(missing_users):
+            profiles_by_userid[userid] = profile
+
+        for userid, score in scores:
+            if profiles_by_userid.get(userid) is None:
+                raise Exception('Logic error, couldn\'t find any profile for {}'.format(userid))
+            profiledata = profiles_by_userid[userid]
+
+            record = Node.void('record')
+            data.add_child(record)
+            record.add_child(Node.u32('mcode', score.id))
+            record.add_child(Node.u8('notetype', self.db_to_game_chart(score.chart)))
+            record.add_child(Node.u8('rank', self.db_to_game_rank(score.data.get_int('rank'))))
+            record.add_child(Node.u8('clearkind', self.db_to_game_halo(score.data.get_int('halo'))))
+            record.add_child(Node.u8('flagdata', 0))
+            record.add_child(Node.string('name', profiledata.get_str('name')))
+            record.add_child(Node.s32('area', profiledata.get_int('area', 58)))
+            record.add_child(Node.s32('code', profiledata.get_int('extid')))
+            record.add_child(Node.s32('score', score.points))
+            record.add_child(Node.s32('ghostid', score.key))
+
+    def __handle_usernew(self, userid: Optional[UserID], requestdata: Node, response: Node) -> None:
+        if userid is None:
+            raise Exception('Expecting valid UserID to create new profile!')
+
+        machine = self.data.local.machine.get_machine(self.config['machine']['pcbid'])
+        self.put_profile(userid, ValidatedDict({
+            'lid': machine.id,
+        }))
+        profile = self.get_profile(userid)
+
+        response.add_child(Node.string('seq', ID.format_extid(profile.get_int('extid'))))
+        response.add_child(Node.s32('code', profile.get_int('extid')))
+        response.add_child(Node.string('shoparea', ''))
+
+    def __handle_inheritance(self, userid: Optional[UserID], requestdata: Node, response: Node) -> None:
+        response.add_child(Node.s32('InheritanceStatus', 0))
+
+    def __handle_ghostload(self, userid: Optional[UserID], requestdata: Node, response: Node) -> None:
+        ghostid = requestdata.child_value('ghostid')
+        ghost = self.data.local.music.get_score_by_key(self.game, self.music_version, ghostid)
+        if ghost is None:
+            return
+
+        userid, score = ghost
+        profile = self.get_profile(userid)
+        if profile is None:
+            return
+
+        if 'ghost' not in score.data:
+            return
+
+        ghostdata = Node.void('ghostdata')
+        response.add_child(ghostdata)
+        ghostdata.add_child(Node.s32('code', profile.get_int('extid')))
+        ghostdata.add_child(Node.u32('mcode', score.id))
+        ghostdata.add_child(Node.u8('notetype', self.db_to_game_chart(score.chart)))
+        ghostdata.add_child(Node.s32('ghostsize', len(score.data['ghost'])))
+        ghostdata.add_child(Node.string('ghost', score.data['ghost']))
+
+    def handle_playerdata_usergamedata_advanced_request(self, request: Node) -> Optional[Node]:
+        playerdata = Node.void('playerdata')
+
+        # DDR Ace decides to be difficult and have a third level of packet switching
+        mode = request.child_value('data/mode')
+        refid = request.child_value('data/refid')
+        extid = request.child_value('data/ddrcode')
+
+        userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        if userid is None:
+            # Possibly look up by extid instead
+            userid = self.data.remote.user.from_extid(self.game, self.version, extid)
+
+        if mode == 'userload':
+            self.__handle_userload(userid, request.child('data'), playerdata)
+        elif mode == 'usersave':
+            self.__handle_usersave(userid, request.child('data'), playerdata)
+        elif mode == 'rivalload':
+            self.__handle_rivalload(userid, request.child('data'), playerdata)
+        elif mode == 'usernew':
+            self.__handle_usernew(userid, request.child('data'), playerdata)
+        elif mode == 'inheritance':
+            self.__handle_inheritance(userid, request.child('data'), playerdata)
+        elif mode == 'ghostload':
+            self.__handle_ghostload(userid, request.child('data'), playerdata)
+        else:
+            # We don't support this
+            return None
+
+        playerdata.add_child(Node.s32('result', 0))
+        return playerdata
+
+    def handle_playerdata_usergamedata_send_request(self, request: Node) -> Node:
+        playerdata = Node.void('playerdata')
+        refid = request.child_value('data/refid')
+
+        userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        if userid is not None:
+            profile = self.get_profile(userid) or ValidatedDict()
+            usergamedata = profile.get_dict('usergamedata')
+
+            for record in request.child('data/record').children:
+                if record.name != 'd':
+                    continue
+
+                strdata = base64.b64decode(record.value)
+                bindata = base64.b64decode(record.child_value('bin1'))
+
+                # Grab and format the profile objects
+                strdatalist = strdata.split(b',')
+                profiletype = strdatalist[1].decode('utf-8')
+                strdatalist = strdatalist[2:]
+
+                # Extract relevant bits for frontend/API
+                if profiletype == 'COMMON':
+                    profile.replace_str('name', strdatalist[self.GAME_COMMON_NAME_OFFSET].decode('ascii'))
+                    profile.replace_int('area', intish(strdatalist[self.GAME_COMMON_AREA_OFFSET].decode('ascii'), 16))
+                    profile.replace_bool('workout_mode', int(strdatalist[self.GAME_COMMON_WEIGHT_DISPLAY_OFFSET].decode('ascii'), 16) != 0)
+                    profile.replace_int('weight', int(float(strdatalist[self.GAME_COMMON_WEIGHT_OFFSET].decode('ascii')) * 10))
+                    profile.replace_int('character', int(strdatalist[self.GAME_COMMON_CHARACTER_OFFSET].decode('ascii'), 16))
+                if profiletype == 'OPTION':
+                    profile.replace_int('combo', int(strdatalist[self.GAME_OPTION_COMBO_POSITION_OFFSET].decode('ascii'), 16))
+                    profile.replace_int('early_late', int(strdatalist[self.GAME_OPTION_FAST_SLOW_OFFSET].decode('ascii'), 16))
+                    profile.replace_int('arrowskin', int(strdatalist[self.GAME_OPTION_ARROW_SKIN_OFFSET].decode('ascii'), 16))
+                    profile.replace_int('guidelines', int(strdatalist[self.GAME_OPTION_GUIDELINE_OFFSET].decode('ascii'), 16))
+                    profile.replace_int('filter', int(strdatalist[self.GAME_OPTION_FILTER_OFFSET].decode('ascii'), 16))
+
+                usergamedata[profiletype] = {
+                    'strdata': b','.join(strdatalist),
+                    'bindata': bindata,
+                }
+
+            profile.replace_dict('usergamedata', usergamedata)
+            self.put_profile(userid, profile)
+
+        playerdata.add_child(Node.s32('result', 0))
+        return playerdata
+
+    def handle_playerdata_usergamedata_recv_request(self, request: Node) -> Node:
+        playerdata = Node.void('playerdata')
+
+        player = Node.void('player')
+        playerdata.add_child(player)
+
+        refid = request.child_value('data/refid')
+        userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        if userid is not None:
+            profile = self.get_profile(userid)
+            links = self.data.local.user.get_links(self.game, self.version, userid)
+            records = 0
+
+            record = Node.void('record')
+            player.add_child(record)
+
+            def acehex(val: int) -> str:
+                return hex(val)[2:]
+
+            if profile is None:
+                # Just return a default empty node
+                record.add_child(Node.string('d', '<NODATA>'))
+                records = 1
+            else:
+                # Figure out what profiles are being requested
+                profiletypes = request.child_value('data/recv_csv').split(',')[::2]
+                usergamedata = profile.get_dict('usergamedata')
+                for ptype in profiletypes:
+                    if ptype in usergamedata:
+                        records = records + 1
+
+                        if ptype == "COMMON":
+                            # Return basic profile options
+                            common = usergamedata[ptype]['strdata'].split(b',')
+                            common[self.GAME_COMMON_NAME_OFFSET] = profile.get_str('name').encode('ascii')
+                            common[self.GAME_COMMON_AREA_OFFSET] = acehex(profile.get_int('area')).encode('ascii')
+                            common[self.GAME_COMMON_WEIGHT_DISPLAY_OFFSET] = b'1' if profile.get_bool('workout_mode') else b'0'
+                            common[self.GAME_COMMON_WEIGHT_OFFSET] = str(float(profile.get_int('weight')) / 10.0).encode('ascii')
+                            common[self.GAME_COMMON_CHARACTER_OFFSET] = acehex(profile.get_int('character')).encode('ascii')
+                            usergamedata[ptype]['strdata'] = b','.join(common)
+                        if ptype == "OPTION":
+                            # Return user settings for frontend
+                            option = usergamedata[ptype]['strdata'].split(b',')
+                            option[self.GAME_OPTION_FAST_SLOW_OFFSET] = acehex(profile.get_int('early_late')).encode('ascii')
+                            option[self.GAME_OPTION_COMBO_POSITION_OFFSET] = acehex(profile.get_int('combo')).encode('ascii')
+                            option[self.GAME_OPTION_ARROW_SKIN_OFFSET] = acehex(profile.get_int('arrowskin')).encode('ascii')
+                            option[self.GAME_OPTION_GUIDELINE_OFFSET] = acehex(profile.get_int('guidelines')).encode('ascii')
+                            option[self.GAME_OPTION_FILTER_OFFSET] = acehex(profile.get_int('filter')).encode('ascii')
+                            usergamedata[ptype]['strdata'] = b','.join(option)
+                        if ptype == "LAST":
+                            # Return the number of calories expended in the last day
+                            workouts = self.data.local.user.get_time_based_achievements(
+                                self.game,
+                                self.version,
+                                userid,
+                                achievementtype='workout',
+                                since=Time.now() - Time.SECONDS_IN_DAY,
+                            )
+                            total = sum([w.data.get_int('calories') for w in workouts])
+
+                            last = usergamedata[ptype]['strdata'].split(b',')
+                            last[self.GAME_LAST_CALORIES_OFFSET] = acehex(total).encode('ascii')
+                            usergamedata[ptype]['strdata'] = b','.join(last)
+                        if ptype == "RIVAL":
+                            # Fill in the DDR code and active status of the three active
+                            # rivals.
+                            rival = usergamedata[ptype]['strdata'].split(b',')
+                            lastdict = profile.get_dict('last')
+
+                            friends: Dict[int, Optional[ValidatedDict]] = {}
+                            for link in links:
+                                if link.type[:7] != 'friend_':
+                                    continue
+
+                                pos = int(link.type[7:])
+                                friends[pos] = self.get_profile(link.other_userid)
+
+                            for rivalno in [1, 2, 3]:
+                                activeslot = {
+                                    1: self.GAME_RIVAL_SLOT_1_ACTIVE_OFFSET,
+                                    2: self.GAME_RIVAL_SLOT_2_ACTIVE_OFFSET,
+                                    3: self.GAME_RIVAL_SLOT_3_ACTIVE_OFFSET,
+                                }[rivalno]
+
+                                whichfriend = lastdict.get_int('rival{}'.format(rivalno)) - 1
+                                if whichfriend < 0:
+                                    # This rival isn't active
+                                    rival[activeslot] = b'0'
+                                    continue
+
+                                friendprofile = friends.get(whichfriend)
+                                if friendprofile is None:
+                                    # This rival doesn't exist
+                                    rival[activeslot] = b'0'
+                                    continue
+
+                                ddrcodeslot = {
+                                    1: self.GAME_RIVAL_SLOT_1_DDRCODE_OFFSET,
+                                    2: self.GAME_RIVAL_SLOT_2_DDRCODE_OFFSET,
+                                    3: self.GAME_RIVAL_SLOT_3_DDRCODE_OFFSET,
+                                }[rivalno]
+
+                                rival[activeslot] = acehex(rivalno).encode('ascii')
+                                rival[ddrcodeslot] = acehex(friendprofile.get_int('extid')).encode('ascii')
+
+                            usergamedata[ptype]['strdata'] = b','.join(rival)
+
+                        dnode = Node.string('d', base64.b64encode(usergamedata[ptype]['strdata']).decode('ascii'))
+                        dnode.add_child(Node.string('bin1', base64.b64encode(usergamedata[ptype]['bindata']).decode('ascii')))
+                        record.add_child(dnode)
+
+            player.add_child(Node.u32('record_num', records))
+
+        playerdata.add_child(Node.s32('result', 0))
+        return playerdata
+
+    def handle_system_convcardnumber_request(self, request: Node) -> Node:
+        cardid = request.child_value('data/card_id')
+        cardnumber = CardCipher.encode(cardid)
+
+        system = Node.void('system')
+        data = Node.void('data')
+        system.add_child(data)
+
+        system.add_child(Node.s32('result', 0))
+        data.add_child(Node.string('card_number', cardnumber))
+        return system

bemani/backend/ddr/ddrx2.py

@@ -0,0 +1,781 @@
+# vim: set fileencoding=utf-8
+import copy
+from typing import Dict, List, Optional, Tuple
+
+from bemani.backend.ddr.base import DDRBase
+from bemani.backend.ddr.stubs import DDRX
+from bemani.backend.ddr.common import (
+    DDRGameFriendHandler,
+    DDRGameLockHandler,
+    DDRGameLoadCourseHandler,
+    DDRGameLoadHandler,
+    DDRGameLogHandler,
+    DDRGameMessageHandler,
+    DDRGameNewHandler,
+    DDRGameOldHandler,
+    DDRGameRankingHandler,
+    DDRGameSaveCourseHandler,
+    DDRGameSaveHandler,
+    DDRGameScoreHandler,
+    DDRGameShopHandler,
+    DDRGameTraceHandler,
+)
+from bemani.common import Time, VersionConstants, ValidatedDict, intish
+from bemani.data import Score, UserID
+from bemani.protocol import Node
+
+
+class DDRX2(
+    DDRGameFriendHandler,
+    DDRGameLockHandler,
+    DDRGameLoadCourseHandler,
+    DDRGameLoadHandler,
+    DDRGameLogHandler,
+    DDRGameMessageHandler,
+    DDRGameOldHandler,
+    DDRGameNewHandler,
+    DDRGameRankingHandler,
+    DDRGameSaveCourseHandler,
+    DDRGameSaveHandler,
+    DDRGameScoreHandler,
+    DDRGameShopHandler,
+    DDRGameTraceHandler,
+    DDRBase,
+):
+
+    name = 'DanceDanceRevolution X2'
+    version = VersionConstants.DDR_X2
+
+    GAME_STYLE_SINGLE = 0
+    GAME_STYLE_DOUBLE = 1
+    GAME_STYLE_VERSUS = 2
+
+    GAME_RANK_AAA = 1
+    GAME_RANK_AA = 2
+    GAME_RANK_A = 3
+    GAME_RANK_B = 4
+    GAME_RANK_C = 5
+    GAME_RANK_D = 6
+    GAME_RANK_E = 7
+
+    GAME_CHART_SINGLE_BEGINNER = 0
+    GAME_CHART_SINGLE_BASIC = 1
+    GAME_CHART_SINGLE_DIFFICULT = 2
+    GAME_CHART_SINGLE_EXPERT = 3
+    GAME_CHART_SINGLE_CHALLENGE = 4
+    GAME_CHART_DOUBLE_BASIC = 5
+    GAME_CHART_DOUBLE_DIFFICULT = 6
+    GAME_CHART_DOUBLE_EXPERT = 7
+    GAME_CHART_DOUBLE_CHALLENGE = 8
+
+    GAME_HALO_NONE = 0
+    GAME_HALO_FULL_COMBO = 1
+    GAME_HALO_PERFECT_COMBO = 2
+    GAME_HALO_MARVELOUS_COMBO = 3
+
+    GAME_MAX_SONGS = 600
+
+    def previous_version(self) -> Optional[DDRBase]:
+        return DDRX(self.data, self.config, self.model)
+
+    def game_to_db_rank(self, game_rank: int) -> int:
+        return {
+            self.GAME_RANK_AAA: self.RANK_AAA,
+            self.GAME_RANK_AA: self.RANK_AA,
+            self.GAME_RANK_A: self.RANK_A,
+            self.GAME_RANK_B: self.RANK_B,
+            self.GAME_RANK_C: self.RANK_C,
+            self.GAME_RANK_D: self.RANK_D,
+            self.GAME_RANK_E: self.RANK_E,
+        }[game_rank]
+
+    def db_to_game_rank(self, db_rank: int) -> int:
+        return {
+            self.RANK_AAA: self.GAME_RANK_AAA,
+            self.RANK_AA_PLUS: self.GAME_RANK_AA,
+            self.RANK_AA: self.GAME_RANK_AA,
+            self.RANK_AA_MINUS: self.GAME_RANK_A,
+            self.RANK_A_PLUS: self.GAME_RANK_A,
+            self.RANK_A: self.GAME_RANK_A,
+            self.RANK_A_MINUS: self.GAME_RANK_B,
+            self.RANK_B_PLUS: self.GAME_RANK_B,
+            self.RANK_B: self.GAME_RANK_B,
+            self.RANK_B_MINUS: self.GAME_RANK_C,
+            self.RANK_C_PLUS: self.GAME_RANK_C,
+            self.RANK_C: self.GAME_RANK_C,
+            self.RANK_C_MINUS: self.GAME_RANK_D,
+            self.RANK_D_PLUS: self.GAME_RANK_D,
+            self.RANK_D: self.GAME_RANK_D,
+            self.RANK_E: self.GAME_RANK_E,
+        }[db_rank]
+
+    def game_to_db_chart(self, game_chart: int) -> int:
+        return {
+            self.GAME_CHART_SINGLE_BEGINNER: self.CHART_SINGLE_BEGINNER,
+            self.GAME_CHART_SINGLE_BASIC: self.CHART_SINGLE_BASIC,
+            self.GAME_CHART_SINGLE_DIFFICULT: self.CHART_SINGLE_DIFFICULT,
+            self.GAME_CHART_SINGLE_EXPERT: self.CHART_SINGLE_EXPERT,
+            self.GAME_CHART_SINGLE_CHALLENGE: self.CHART_SINGLE_CHALLENGE,
+            self.GAME_CHART_DOUBLE_BASIC: self.CHART_DOUBLE_BASIC,
+            self.GAME_CHART_DOUBLE_DIFFICULT: self.CHART_DOUBLE_DIFFICULT,
+            self.GAME_CHART_DOUBLE_EXPERT: self.CHART_DOUBLE_EXPERT,
+            self.GAME_CHART_DOUBLE_CHALLENGE: self.CHART_DOUBLE_CHALLENGE,
+        }[game_chart]
+
+    def db_to_game_chart(self, db_chart: int) -> int:
+        return {
+            self.CHART_SINGLE_BEGINNER: self.GAME_CHART_SINGLE_BEGINNER,
+            self.CHART_SINGLE_BASIC: self.GAME_CHART_SINGLE_BASIC,
+            self.CHART_SINGLE_DIFFICULT: self.GAME_CHART_SINGLE_DIFFICULT,
+            self.CHART_SINGLE_EXPERT: self.GAME_CHART_SINGLE_EXPERT,
+            self.CHART_SINGLE_CHALLENGE: self.GAME_CHART_SINGLE_CHALLENGE,
+            self.CHART_DOUBLE_BASIC: self.GAME_CHART_DOUBLE_BASIC,
+            self.CHART_DOUBLE_DIFFICULT: self.GAME_CHART_DOUBLE_DIFFICULT,
+            self.CHART_DOUBLE_EXPERT: self.GAME_CHART_DOUBLE_EXPERT,
+            self.CHART_DOUBLE_CHALLENGE: self.GAME_CHART_DOUBLE_CHALLENGE,
+        }[db_chart]
+
+    def db_to_game_halo(self, db_halo: int) -> int:
+        if db_halo == self.HALO_MARVELOUS_FULL_COMBO:
+            combo_type = self.GAME_HALO_MARVELOUS_COMBO
+        elif db_halo == self.HALO_PERFECT_FULL_COMBO:
+            combo_type = self.GAME_HALO_PERFECT_COMBO
+        elif db_halo == self.HALO_GREAT_FULL_COMBO:
+            combo_type = self.GAME_HALO_FULL_COMBO
+        else:
+            combo_type = self.GAME_HALO_NONE
+        return combo_type
+
+    def handle_game_common_request(self, request: Node) -> Node:
+        game = Node.void('game')
+        for flagid in range(256):
+            flag = Node.void('flag')
+            game.add_child(flag)
+
+            flag.set_attribute('id', str(flagid))
+            flag.set_attribute('s2', '0')
+            flag.set_attribute('s1', '0')
+            flag.set_attribute('t', '0')
+
+        hit_chart = self.data.local.music.get_hit_chart(self.game, self.music_version, self.GAME_MAX_SONGS)
+        counts_by_reflink = [0] * self.GAME_MAX_SONGS
+        for (reflink, plays) in hit_chart:
+            if reflink >= 0 and reflink < self.GAME_MAX_SONGS:
+                counts_by_reflink[reflink] = plays
+        game.add_child(Node.u32_array('cnt_music', counts_by_reflink))
+
+        return game
+
+    def handle_game_hiscore_request(self, request: Node) -> Node:
+        # This is almost identical to X3 and above, except X3 added a 'code' field
+        # that isn't present here. In the interest of correctness, keep a separate
+        # implementation here.
+        records = self.data.remote.music.get_all_records(self.game, self.music_version)
+
+        sortedrecords: Dict[int, Dict[int, Tuple[UserID, Score]]] = {}
+        missing_profiles = []
+        for (userid, score) in records:
+            if score.id not in sortedrecords:
+                sortedrecords[score.id] = {}
+            sortedrecords[score.id][score.chart] = (userid, score)
+            missing_profiles.append(userid)
+        users = {userid: profile for (userid, profile) in self.get_any_profiles(missing_profiles)}
+
+        game = Node.void('game')
+        for song in sortedrecords:
+            music = Node.void('music')
+            game.add_child(music)
+            music.set_attribute('reclink_num', str(song))
+
+            for chart in sortedrecords[song]:
+                userid, score = sortedrecords[song][chart]
+                try:
+                    gamechart = self.db_to_game_chart(chart)
+                except KeyError:
+                    # Don't support this chart in this game
+                    continue
+                gamerank = self.db_to_game_rank(score.data.get_int('rank'))
+                combo_type = self.db_to_game_halo(score.data.get_int('halo'))
+
+                typenode = Node.void('type')
+                music.add_child(typenode)
+                typenode.set_attribute('diff', str(gamechart))
+
+                typenode.add_child(Node.string('name', users[userid].get_str('name')))
+                typenode.add_child(Node.u32('score', score.points))
+                typenode.add_child(Node.u16('area', users[userid].get_int('area', 51)))
+                typenode.add_child(Node.u8('rank', gamerank))
+                typenode.add_child(Node.u8('combo_type', combo_type))
+
+        return game
+
+    def handle_game_load_m_request(self, request: Node) -> Node:
+        extid = intish(request.attribute('code'))
+        refid = request.attribute('refid')
+
+        if extid is not None:
+            # Rival score loading
+            userid = self.data.remote.user.from_extid(self.game, self.version, extid)
+        else:
+            # Self score loading
+            userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+
+        if userid is not None:
+            scores = self.data.remote.music.get_scores(self.game, self.music_version, userid)
+        else:
+            scores = []
+
+        sortedscores: Dict[int, Dict[int, Score]] = {}
+        for score in scores:
+            if score.id not in sortedscores:
+                sortedscores[score.id] = {}
+            sortedscores[score.id][score.chart] = score
+
+        game = Node.void('game')
+        for song in sortedscores:
+            music = Node.void('music')
+            game.add_child(music)
+            music.set_attribute('reclink', str(song))
+
+            for chart in sortedscores[song]:
+                score = sortedscores[song][chart]
+                try:
+                    gamechart = self.db_to_game_chart(chart)
+                except KeyError:
+                    # Don't support this chart in this game
+                    continue
+                gamerank = self.db_to_game_rank(score.data.get_int('rank'))
+                combo_type = self.db_to_game_halo(score.data.get_int('halo'))
+
+                typenode = Node.void('type')
+                music.add_child(typenode)
+                typenode.set_attribute('diff', str(gamechart))
+
+                typenode.add_child(Node.u32('score', score.points))
+                typenode.add_child(Node.u16('count', score.plays))
+                typenode.add_child(Node.u8('rank', gamerank))
+                typenode.add_child(Node.u8('combo_type', combo_type))
+
+        return game
+
+    def handle_game_save_m_request(self, request: Node) -> Node:
+        refid = request.attribute('refid')
+        songid = int(request.attribute('mid'))
+        chart = self.game_to_db_chart(int(request.attribute('mtype')))
+
+        # Calculate statistics
+        data = request.child('data')
+        points = int(data.attribute('score'))
+        combo = int(data.attribute('combo'))
+        rank = self.game_to_db_rank(int(data.attribute('rank')))
+        if int(data.attribute('full')) == 0:
+            halo = self.HALO_NONE
+        elif int(data.attribute('perf')) == 0:
+            halo = self.HALO_GREAT_FULL_COMBO
+        elif points < 1000000:
+            halo = self.HALO_PERFECT_FULL_COMBO
+        else:
+            halo = self.HALO_MARVELOUS_FULL_COMBO
+        trace = request.child_value('trace')
+
+        # Save the score, regardless of whether we have a refid. If we save
+        # an anonymous score, it only goes into the DB to count against the
+        # number of plays for that song/chart.
+        userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        self.update_score(
+            userid,
+            songid,
+            chart,
+            points,
+            rank,
+            halo,
+            combo,
+            trace,
+        )
+
+        # No response needed
+        game = Node.void('game')
+        return game
+
+    def format_profile(self, userid: UserID, profile: ValidatedDict) -> Node:
+        root = Node.void('game')
+
+        # Look up play stats we bridge to every mix
+        play_stats = self.get_play_statistics(userid)
+
+        # Basic game settings
+        root.add_child(Node.string('seq', ''))
+        root.add_child(Node.u32('code', profile.get_int('extid')))
+        root.add_child(Node.string('name', profile.get_str('name')))
+        root.add_child(Node.u8('area', profile.get_int('area', 51)))
+        root.add_child(Node.u32('cnt_s', play_stats.get_int('single_plays')))
+        root.add_child(Node.u32('cnt_d', play_stats.get_int('double_plays')))
+        root.add_child(Node.u32('cnt_b', play_stats.get_int('battle_plays')))  # This could be wrong, its a guess
+        root.add_child(Node.u32('cnt_m0', play_stats.get_int('cnt_m0')))
+        root.add_child(Node.u32('cnt_m1', play_stats.get_int('cnt_m1')))
+        root.add_child(Node.u32('cnt_m2', play_stats.get_int('cnt_m2')))
+        root.add_child(Node.u32('cnt_m3', play_stats.get_int('cnt_m3')))
+        root.add_child(Node.u32('exp', play_stats.get_int('exp')))
+        root.add_child(Node.u32('exp_o', profile.get_int('exp_o')))
+        root.add_child(Node.u32('star', profile.get_int('star')))
+        root.add_child(Node.u32('star_c', profile.get_int('star_c')))
+        root.add_child(Node.u8('combo', profile.get_int('combo', 0)))
+        root.add_child(Node.u8('timing_diff', profile.get_int('early_late', 0)))
+
+        # Character stuff
+        chara = Node.void('chara')
+        root.add_child(chara)
+        if 'chara' in profile:
+            chara.set_attribute('my', str(profile.get_int('chara')))
+
+        root.add_child(Node.u8_array('chara_opt', profile.get_int_array('chara_opt', 96)))
+
+        # Drill rankings
+        if 'title' in profile:
+            title = Node.void('title')
+            root.add_child(title)
+            titledict = profile.get_dict('title')
+            if 't' in titledict:
+                title.set_attribute('t', str(titledict.get_int('t')))
+            if 's' in titledict:
+                title.set_attribute('s', str(titledict.get_int('s')))
+            if 'd' in titledict:
+                title.set_attribute('d', str(titledict.get_int('d')))
+
+        if 'title_gr' in profile:
+            title_gr = Node.void('title_gr')
+            root.add_child(title_gr)
+            title_grdict = profile.get_dict('title_gr')
+            if 't' in title_grdict:
+                title_gr.set_attribute('t', str(title_grdict.get_int('t')))
+            if 's' in title_grdict:
+                title_gr.set_attribute('s', str(title_grdict.get_int('s')))
+            if 'd' in title_grdict:
+                title_gr.set_attribute('d', str(title_grdict.get_int('d')))
+
+        # Event progrses
+        if 'event' in profile:
+            event = Node.void('event')
+            root.add_child(event)
+            event_dict = profile.get_dict('event')
+            if 'diff_sum' in event_dict:
+                event.set_attribute('diff_sum', str(event_dict.get_int('diff_sum')))
+            if 'welcome' in event_dict:
+                event.set_attribute('welcome', str(event_dict.get_int('welcome')))
+            if 'e_flags' in event_dict:
+                event.set_attribute('e_flags', str(event_dict.get_int('e_flags')))
+
+        if 'e_panel' in profile:
+            e_panel = Node.void('e_panel')
+            root.add_child(e_panel)
+            e_panel_dict = profile.get_dict('e_panel')
+            if 'play_id' in e_panel_dict:
+                e_panel.set_attribute('play_id', str(e_panel_dict.get_int('play_id')))
+            e_panel.add_child(Node.u8_array('cell', e_panel_dict.get_int_array('cell', 24)))
+            e_panel.add_child(Node.u8_array('panel_state', e_panel_dict.get_int_array('panel_state', 6)))
+
+        if 'e_pix' in profile:
+            e_pix = Node.void('e_pix')
+            root.add_child(e_pix)
+            e_pix_dict = profile.get_dict('e_pix')
+            if 'max_distance' in e_pix_dict:
+                e_pix.set_attribute('max_distance', str(e_pix_dict.get_int('max_distance')))
+            if 'max_planet' in e_pix_dict:
+                e_pix.set_attribute('max_planet', str(e_pix_dict.get_int('max_planet')))
+            if 'total_distance' in e_pix_dict:
+                e_pix.set_attribute('total_distance', str(e_pix_dict.get_int('total_distance')))
+            if 'total_planet' in e_pix_dict:
+                e_pix.set_attribute('total_planet', str(e_pix_dict.get_int('total_planet')))
+            if 'border_character' in e_pix_dict:
+                e_pix.set_attribute('border_character', str(e_pix_dict.get_int('border_character')))
+            if 'border_balloon' in e_pix_dict:
+                e_pix.set_attribute('border_balloon', str(e_pix_dict.get_int('border_balloon')))
+            if 'border_music_aftr' in e_pix_dict:
+                e_pix.set_attribute('border_music_aftr', str(e_pix_dict.get_int('border_music_aftr')))
+            if 'border_music_meii' in e_pix_dict:
+                e_pix.set_attribute('border_music_meii', str(e_pix_dict.get_int('border_music_meii')))
+            if 'border_music_dirt' in e_pix_dict:
+                e_pix.set_attribute('border_music_dirt', str(e_pix_dict.get_int('border_music_dirt')))
+            if 'flags' in e_pix_dict:
+                e_pix.set_attribute('flags', str(e_pix_dict.get_int('flags')))
+
+        # Calorie mode
+        if 'weight' in profile:
+            workouts = self.data.local.user.get_time_based_achievements(
+                self.game,
+                self.version,
+                userid,
+                achievementtype='workout',
+                since=Time.now() - Time.SECONDS_IN_DAY,
+            )
+            total = sum([w.data.get_int('calories') for w in workouts])
+            workout = Node.void('workout')
+            root.add_child(workout)
+            workout.set_attribute('weight', str(profile.get_int('weight')))
+            workout.set_attribute('day', str(total))
+            workout.set_attribute('disp', '1')
+
+        # Last cursor settings
+        last = Node.void('last')
+        root.add_child(last)
+        lastdict = profile.get_dict('last')
+        last.set_attribute('fri', str(lastdict.get_int('fri')))
+        last.set_attribute('style', str(lastdict.get_int('style')))
+        last.set_attribute('mode', str(lastdict.get_int('mode')))
+        last.set_attribute('cate', str(lastdict.get_int('cate')))
+        last.set_attribute('sort', str(lastdict.get_int('sort')))
+        last.set_attribute('mid', str(lastdict.get_int('mid')))
+        last.set_attribute('mtype', str(lastdict.get_int('mtype')))
+        last.set_attribute('cid', str(lastdict.get_int('cid')))
+        last.set_attribute('ctype', str(lastdict.get_int('ctype')))
+        last.set_attribute('sid', str(lastdict.get_int('sid')))
+
+        # Groove gauge level-ups
+        gr_s = Node.void('gr_s')
+        root.add_child(gr_s)
+        index = 1
+        for entry in profile.get_int_array('gr_s', 5):
+            gr_s.set_attribute('gr{}'.format(index), str(entry))
+            index = index + 1
+
+        gr_d = Node.void('gr_d')
+        root.add_child(gr_d)
+        index = 1
+        for entry in profile.get_int_array('gr_d', 5):
+            gr_d.set_attribute('gr{}'.format(index), str(entry))
+            index = index + 1
+
+        # Options in menus
+        root.add_child(Node.s16_array('opt', profile.get_int_array('opt', 16)))
+        root.add_child(Node.s16_array('opt_ex', profile.get_int_array('opt_ex', 16)))
+
+        # Unlock flags
+        root.add_child(Node.u8_array('flag', profile.get_int_array('flag', 256, [1] * 256)))
+
+        # Ranking display?
+        root.add_child(Node.u16_array('rank', profile.get_int_array('rank', 100)))
+
+        # Rivals
+        links = self.data.local.user.get_links(self.game, self.version, userid)
+        for link in links:
+            if link.type[:7] != 'friend_':
+                continue
+
+            pos = int(link.type[7:])
+            friend = self.get_profile(link.other_userid)
+            play_stats = self.get_play_statistics(link.other_userid)
+            if friend is not None:
+                friendnode = Node.void('friend')
+                root.add_child(friendnode)
+                friendnode.set_attribute('pos', str(pos))
+                friendnode.set_attribute('vs', '0')
+                friendnode.set_attribute('up', '0')
+                friendnode.add_child(Node.u32('code', friend.get_int('extid')))
+                friendnode.add_child(Node.string('name', friend.get_str('name')))
+                friendnode.add_child(Node.u8('area', friend.get_int('area', 51)))
+                friendnode.add_child(Node.u32('exp', play_stats.get_int('exp')))
+                friendnode.add_child(Node.u32('star', friend.get_int('star')))
+
+                # Drill rankings
+                if 'title' in friend:
+                    title = Node.void('title')
+                    friendnode.add_child(title)
+                    titledict = friend.get_dict('title')
+                    if 't' in titledict:
+                        title.set_attribute('t', str(titledict.get_int('t')))
+                    if 's' in titledict:
+                        title.set_attribute('s', str(titledict.get_int('s')))
+                    if 'd' in titledict:
+                        title.set_attribute('d', str(titledict.get_int('d')))
+
+                if 'title_gr' in friend:
+                    title_gr = Node.void('title_gr')
+                    friendnode.add_child(title_gr)
+                    title_grdict = friend.get_dict('title_gr')
+                    if 't' in title_grdict:
+                        title_gr.set_attribute('t', str(title_grdict.get_int('t')))
+                    if 's' in title_grdict:
+                        title_gr.set_attribute('s', str(title_grdict.get_int('s')))
+                    if 'd' in title_grdict:
+                        title_gr.set_attribute('d', str(title_grdict.get_int('d')))
+
+                # Groove gauge level-ups
+                gr_s = Node.void('gr_s')
+                friendnode.add_child(gr_s)
+                index = 1
+                for entry in friend.get_int_array('gr_s', 5):
+                    gr_s.set_attribute('gr{}'.format(index), str(entry))
+                    index = index + 1
+
+                gr_d = Node.void('gr_d')
+                friendnode.add_child(gr_d)
+                index = 1
+                for entry in friend.get_int_array('gr_d', 5):
+                    gr_d.set_attribute('gr{}'.format(index), str(entry))
+                    index = index + 1
+
+        return root
+
+    def unformat_profile(self, userid: UserID, request: Node, oldprofile: ValidatedDict) -> ValidatedDict:
+        newprofile = copy.deepcopy(oldprofile)
+        play_stats = self.get_play_statistics(userid)
+
+        # Grab last node and accessories so we can make decisions based on type
+        last = request.child('last')
+        lastdict = newprofile.get_dict('last')
+        mode = int(last.attribute('mode'))
+        style = int(last.attribute('style'))
+        is_dp = style == self.GAME_STYLE_DOUBLE
+
+        # Drill rankings
+        title = request.child('title')
+        title_gr = request.child('title_gr')
+        titledict = newprofile.get_dict('title')
+        title_grdict = newprofile.get_dict('title_gr')
+
+        # Groove radar level ups
+        gr = request.child('gr')
+
+        # Set the correct values depending on if we're single or double play
+        if is_dp:
+            play_stats.increment_int('double_plays')
+            if gr is not None:
+                newprofile.replace_int_array(
+                    'gr_d',
+                    5,
+                    [
+                        intish(gr.attribute('gr1')),
+                        intish(gr.attribute('gr2')),
+                        intish(gr.attribute('gr3')),
+                        intish(gr.attribute('gr4')),
+                        intish(gr.attribute('gr5')),
+                    ],
+                )
+            if title is not None:
+                titledict.replace_int('d', title.value)
+                newprofile.replace_dict('title', titledict)
+            if title_gr is not None:
+                title_grdict.replace_int('d', title.value)
+                newprofile.replace_dict('title_gr', title_grdict)
+        else:
+            play_stats.increment_int('single_plays')
+            if gr is not None:
+                newprofile.replace_int_array(
+                    'gr_s',
+                    5,
+                    [
+                        intish(gr.attribute('gr1')),
+                        intish(gr.attribute('gr2')),
+                        intish(gr.attribute('gr3')),
+                        intish(gr.attribute('gr4')),
+                        intish(gr.attribute('gr5')),
+                    ],
+                )
+            if title is not None:
+                titledict.replace_int('s', title.value)
+                newprofile.replace_dict('title', titledict)
+            if title_gr is not None:
+                title_grdict.replace_int('s', title.value)
+                newprofile.replace_dict('title_gr', title_grdict)
+        play_stats.increment_int('cnt_m{}'.format(mode))
+
+        # Update last attributes
+        lastdict.replace_int('fri', intish(last.attribute('fri')))
+        lastdict.replace_int('style', intish(last.attribute('style')))
+        lastdict.replace_int('mode', intish(last.attribute('mode')))
+        lastdict.replace_int('cate', intish(last.attribute('cate')))
+        lastdict.replace_int('sort', intish(last.attribute('sort')))
+        lastdict.replace_int('mid', intish(last.attribute('mid')))
+        lastdict.replace_int('mtype', intish(last.attribute('mtype')))
+        lastdict.replace_int('cid', intish(last.attribute('cid')))
+        lastdict.replace_int('ctype', intish(last.attribute('ctype')))
+        lastdict.replace_int('sid', intish(last.attribute('sid')))
+        newprofile.replace_dict('last', lastdict)
+
+        # Grab character options
+        chara = request.child('chara')
+        if chara is not None:
+            newprofile.replace_int('chara', intish(chara.attribute('my')))
+        newprofile.replace_int_array('chara_opt', 96, request.child_value('chara_opt'))
+
+        # Track event progress
+        event = request.child('event')
+        if event is not None:
+            event_dict = newprofile.get_dict('event')
+            event_dict.replace_int('diff_sum', intish(event.attribute('diff_sum')))
+            event_dict.replace_int('e_flags', intish(event.attribute('e_flags')))
+            event_dict.replace_int('welcome', intish(event.attribute('welcome')))
+            newprofile.replace_dict('event', event_dict)
+
+        e_panel = request.child('e_panel')
+        if e_panel is not None:
+            e_panel_dict = newprofile.get_dict('e_panel')
+            e_panel_dict.replace_int('play_id', intish(e_panel.attribute('play_id')))
+            e_panel_dict.replace_int_array('cell', 24, e_panel.child_value('cell'))
+            e_panel_dict.replace_int_array('panel_state', 6, e_panel.child_value('panel_state'))
+            newprofile.replace_dict('e_panel', e_panel_dict)
+
+        e_pix = request.child('e_pix')
+        if e_pix is not None:
+            e_pix_dict = newprofile.get_dict('e_pix')
+            max_distance = e_pix_dict.get_int('max_distance')
+            max_planet = e_pix_dict.get_int('max_planet')
+            total_distance = e_pix_dict.get_int('total_distance')
+            total_planet = e_pix_dict.get_int('total_planet')
+            cur_distance = intish(e_pix.attribute('this_distance'))
+            cur_planet = intish(e_pix.attribute('this_planet'))
+            if cur_distance is not None:
+                max_distance = max(max_distance, cur_distance)
+                total_distance += cur_distance
+            if cur_planet is not None:
+                max_planet = max(max_planet, cur_planet)
+                total_planet += cur_planet
+
+            e_pix_dict.replace_int('max_distance', max_distance)
+            e_pix_dict.replace_int('max_planet', max_planet)
+            e_pix_dict.replace_int('total_distance', total_distance)
+            e_pix_dict.replace_int('total_planet', total_planet)
+            e_pix_dict.replace_int('flags', intish(e_pix.attribute('flags')))
+            newprofile.replace_dict('e_pix', e_pix_dict)
+
+        # Options
+        opt = request.child('opt')
+        if opt is not None:
+            # A bug in old versions of AVS returns the wrong number for set
+            newprofile.replace_int_array('opt', 16, opt.value[:16])
+
+        # Experience and stars
+        exp = request.child_value('exp')
+        if exp is not None:
+            play_stats.replace_int('exp', play_stats.get_int('exp') + exp)
+        star = request.child_value('star')
+        if star is not None:
+            newprofile.replace_int('star', newprofile.get_int('star') + star)
+        star_c = request.child_value('star_c')
+        if star_c is not None:
+            newprofile.replace_int('star_c', newprofile.get_int('star_c') + exp)
+
+        # Update game flags
+        for child in request.children:
+            if child.name != 'flag':
+                continue
+            try:
+                value = int(child.attribute('data'))
+                offset = int(child.attribute('no'))
+            except ValueError:
+                continue
+
+            flags = newprofile.get_int_array('flag', 256, [1] * 256)
+            if offset < 0 or offset >= len(flags):
+                continue
+            flags[offset] = value
+            newprofile.replace_int_array('flag', 256, flags)
+
+        # Workout mode support
+        newweight = -1
+        oldweight = newprofile.get_int('weight')
+        for child in request.children:
+            if child.name != 'weight':
+                continue
+            newweight = child.value
+        if newweight < 0:
+            newweight = oldweight
+
+        # Either update or unset the weight depending on the game
+        if newweight == 0:
+            # Weight is unset or we declined to use this feature, remove from profile
+            if 'weight' in newprofile:
+                del newprofile['weight']
+        else:
+            # Weight has been set or previously retrieved, we should save calories
+            newprofile.replace_int('weight', newweight)
+            total = 0
+            for child in request.children:
+                if child.name != 'calory':
+                    continue
+                total += child.value
+            self.data.local.user.put_time_based_achievement(
+                self.game,
+                self.version,
+                userid,
+                0,
+                'workout',
+                {
+                    'calories': total,
+                    'weight': newweight,
+                },
+            )
+
+        # Look up old friends
+        oldfriends: List[Optional[UserID]] = [None] * 10
+        links = self.data.local.user.get_links(self.game, self.version, userid)
+        for link in links:
+            if link.type[:7] != 'friend_':
+                continue
+
+            pos = int(link.type[7:])
+            oldfriends[pos] = link.other_userid
+
+        # Save any rivals that were added/removed/changed
+        newfriends = oldfriends[:]
+        for child in request.children:
+            if child.name != 'friend':
+                continue
+
+            code = int(child.attribute('code'))
+            pos = int(child.attribute('pos'))
+
+            if pos >= 0 and pos < 10:
+                if code == 0:
+                    # We cleared this friend
+                    newfriends[pos] = None
+                else:
+                    # Try looking up the userid
+                    newfriends[pos] = self.data.remote.user.from_extid(self.game, self.version, code)
+
+        # Diff the set of links to determine updates
+        for i in range(10):
+            if newfriends[i] == oldfriends[i]:
+                continue
+
+            if newfriends[i] is None:
+                # Kill the rival in this location
+                self.data.local.user.destroy_link(
+                    self.game,
+                    self.version,
+                    userid,
+                    'friend_{}'.format(i),
+                    oldfriends[i],
+                )
+            elif oldfriends[i] is None:
+                # Add rival in this location
+                self.data.local.user.put_link(
+                    self.game,
+                    self.version,
+                    userid,
+                    'friend_{}'.format(i),
+                    newfriends[i],
+                    {},
+                )
+            else:
+                # Changed the rival here, kill the old one, add the new one
+                self.data.local.user.destroy_link(
+                    self.game,
+                    self.version,
+                    userid,
+                    'friend_{}'.format(i),
+                    oldfriends[i],
+                )
+                self.data.local.user.put_link(
+                    self.game,
+                    self.version,
+                    userid,
+                    'friend_{}'.format(i),
+                    newfriends[i],
+                    {},
+                )
+
+        # Keep track of play statistics
+        self.update_play_statistics(userid, play_stats)
+
+        return newprofile

bemani/backend/ddr/ddrx3.py

@@ -0,0 +1,814 @@
+# vim: set fileencoding=utf-8
+import copy
+from typing import Dict, List, Optional, Union
+
+from bemani.backend.ddr.base import DDRBase
+from bemani.backend.ddr.ddrx2 import DDRX2
+from bemani.backend.ddr.common import (
+    DDRGameAreaHiscoreHandler,
+    DDRGameFriendHandler,
+    DDRGameHiscoreHandler,
+    DDRGameLoadCourseHandler,
+    DDRGameLoadDailyHandler,
+    DDRGameLoadHandler,
+    DDRGameLockHandler,
+    DDRGameLogHandler,
+    DDRGameMessageHandler,
+    DDRGameNewHandler,
+    DDRGameOldHandler,
+    DDRGameRankingHandler,
+    DDRGameSaveCourseHandler,
+    DDRGameSaveHandler,
+    DDRGameScoreHandler,
+    DDRGameShopHandler,
+    DDRGameTraceHandler,
+)
+from bemani.common import ValidatedDict, VersionConstants, Time, intish
+from bemani.data import Achievement, Score, UserID
+from bemani.protocol import Node
+
+
+class DDRX3(
+    DDRGameAreaHiscoreHandler,
+    DDRGameFriendHandler,
+    DDRGameHiscoreHandler,
+    DDRGameLockHandler,
+    DDRGameLoadCourseHandler,
+    DDRGameLoadDailyHandler,
+    DDRGameLoadHandler,
+    DDRGameLogHandler,
+    DDRGameMessageHandler,
+    DDRGameNewHandler,
+    DDRGameOldHandler,
+    DDRGameRankingHandler,
+    DDRGameSaveCourseHandler,
+    DDRGameSaveHandler,
+    DDRGameScoreHandler,
+    DDRGameShopHandler,
+    DDRGameTraceHandler,
+    DDRBase,
+):
+
+    name = 'DanceDanceRevolution X3 VS 2ndMIX'
+    version = VersionConstants.DDR_X3_VS_2NDMIX
+
+    GAME_STYLE_SINGLE = 0
+    GAME_STYLE_DOUBLE = 1
+    GAME_STYLE_VERSUS = 2
+
+    GAME_RANK_AAA = 1
+    GAME_RANK_AA = 2
+    GAME_RANK_A = 3
+    GAME_RANK_B = 4
+    GAME_RANK_C = 5
+    GAME_RANK_D = 6
+    GAME_RANK_E = 7
+
+    GAME_CHART_SINGLE_BEGINNER = 0
+    GAME_CHART_SINGLE_BASIC = 1
+    GAME_CHART_SINGLE_DIFFICULT = 2
+    GAME_CHART_SINGLE_EXPERT = 3
+    GAME_CHART_SINGLE_CHALLENGE = 4
+    GAME_CHART_DOUBLE_BASIC = 5
+    GAME_CHART_DOUBLE_DIFFICULT = 6
+    GAME_CHART_DOUBLE_EXPERT = 7
+    GAME_CHART_DOUBLE_CHALLENGE = 8
+
+    GAME_HALO_NONE = 0
+    GAME_HALO_FULL_COMBO = 1
+    GAME_HALO_PERFECT_COMBO = 2
+    GAME_HALO_MARVELOUS_COMBO = 3
+
+    GAME_PLAY_MODE_NORMAL = 1
+    GAME_PLAY_MODE_2NDMIX = 5
+
+    GAME_MAX_SONGS = 700
+
+    def previous_version(self) -> Optional[DDRBase]:
+        return DDRX2(self.data, self.config, self.model)
+
+    def game_to_db_rank(self, game_rank: int) -> int:
+        return {
+            self.GAME_RANK_AAA: self.RANK_AAA,
+            self.GAME_RANK_AA: self.RANK_AA,
+            self.GAME_RANK_A: self.RANK_A,
+            self.GAME_RANK_B: self.RANK_B,
+            self.GAME_RANK_C: self.RANK_C,
+            self.GAME_RANK_D: self.RANK_D,
+            self.GAME_RANK_E: self.RANK_E,
+        }[game_rank]
+
+    def db_to_game_rank(self, db_rank: int) -> int:
+        return {
+            self.RANK_AAA: self.GAME_RANK_AAA,
+            self.RANK_AA_PLUS: self.GAME_RANK_AA,
+            self.RANK_AA: self.GAME_RANK_AA,
+            self.RANK_AA_MINUS: self.GAME_RANK_A,
+            self.RANK_A_PLUS: self.GAME_RANK_A,
+            self.RANK_A: self.GAME_RANK_A,
+            self.RANK_A_MINUS: self.GAME_RANK_B,
+            self.RANK_B_PLUS: self.GAME_RANK_B,
+            self.RANK_B: self.GAME_RANK_B,
+            self.RANK_B_MINUS: self.GAME_RANK_C,
+            self.RANK_C_PLUS: self.GAME_RANK_C,
+            self.RANK_C: self.GAME_RANK_C,
+            self.RANK_C_MINUS: self.GAME_RANK_D,
+            self.RANK_D_PLUS: self.GAME_RANK_D,
+            self.RANK_D: self.GAME_RANK_D,
+            self.RANK_E: self.GAME_RANK_E,
+        }[db_rank]
+
+    def game_to_db_chart(self, game_chart: int) -> int:
+        return {
+            self.GAME_CHART_SINGLE_BEGINNER: self.CHART_SINGLE_BEGINNER,
+            self.GAME_CHART_SINGLE_BASIC: self.CHART_SINGLE_BASIC,
+            self.GAME_CHART_SINGLE_DIFFICULT: self.CHART_SINGLE_DIFFICULT,
+            self.GAME_CHART_SINGLE_EXPERT: self.CHART_SINGLE_EXPERT,
+            self.GAME_CHART_SINGLE_CHALLENGE: self.CHART_SINGLE_CHALLENGE,
+            self.GAME_CHART_DOUBLE_BASIC: self.CHART_DOUBLE_BASIC,
+            self.GAME_CHART_DOUBLE_DIFFICULT: self.CHART_DOUBLE_DIFFICULT,
+            self.GAME_CHART_DOUBLE_EXPERT: self.CHART_DOUBLE_EXPERT,
+            self.GAME_CHART_DOUBLE_CHALLENGE: self.CHART_DOUBLE_CHALLENGE,
+        }[game_chart]
+
+    def db_to_game_chart(self, db_chart: int) -> int:
+        return {
+            self.CHART_SINGLE_BEGINNER: self.GAME_CHART_SINGLE_BEGINNER,
+            self.CHART_SINGLE_BASIC: self.GAME_CHART_SINGLE_BASIC,
+            self.CHART_SINGLE_DIFFICULT: self.GAME_CHART_SINGLE_DIFFICULT,
+            self.CHART_SINGLE_EXPERT: self.GAME_CHART_SINGLE_EXPERT,
+            self.CHART_SINGLE_CHALLENGE: self.GAME_CHART_SINGLE_CHALLENGE,
+            self.CHART_DOUBLE_BASIC: self.GAME_CHART_DOUBLE_BASIC,
+            self.CHART_DOUBLE_DIFFICULT: self.GAME_CHART_DOUBLE_DIFFICULT,
+            self.CHART_DOUBLE_EXPERT: self.GAME_CHART_DOUBLE_EXPERT,
+            self.CHART_DOUBLE_CHALLENGE: self.GAME_CHART_DOUBLE_CHALLENGE,
+        }[db_chart]
+
+    def db_to_game_halo(self, db_halo: int) -> int:
+        if db_halo == self.HALO_MARVELOUS_FULL_COMBO:
+            combo_type = self.GAME_HALO_MARVELOUS_COMBO
+        elif db_halo == self.HALO_PERFECT_FULL_COMBO:
+            combo_type = self.GAME_HALO_PERFECT_COMBO
+        elif db_halo == self.HALO_GREAT_FULL_COMBO:
+            combo_type = self.GAME_HALO_FULL_COMBO
+        else:
+            combo_type = self.GAME_HALO_NONE
+        return combo_type
+
+    def handle_game_common_request(self, request: Node) -> Node:
+        game = Node.void('game')
+        for flagid in range(256):
+            flag = Node.void('flag')
+            game.add_child(flag)
+
+            flag.set_attribute('id', str(flagid))
+            flag.set_attribute('s2', '0')
+            flag.set_attribute('s1', '0')
+            flag.set_attribute('t', '0')
+
+        hit_chart = self.data.local.music.get_hit_chart(self.game, self.music_version, self.GAME_MAX_SONGS)
+        counts_by_reflink = [0] * self.GAME_MAX_SONGS
+        for (reflink, plays) in hit_chart:
+            if reflink >= 0 and reflink < self.GAME_MAX_SONGS:
+                counts_by_reflink[reflink] = plays
+        game.add_child(Node.u32_array('cnt_music', counts_by_reflink))
+
+        return game
+
+    def handle_game_load_m_request(self, request: Node) -> Node:
+        extid = intish(request.attribute('code'))
+        refid = request.attribute('refid')
+
+        if extid is not None:
+            # Rival score loading
+            userid = self.data.remote.user.from_extid(self.game, self.version, extid)
+        else:
+            # Self score loading
+            userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+
+        if userid is not None:
+            scores = self.data.remote.music.get_scores(self.game, self.music_version, userid)
+            old_scores = [
+                score for score in self.data.local.user.get_achievements(self.game, self.music_version, userid)
+                if score.type == '2ndmix'
+            ]
+        else:
+            scores = []
+            old_scores = []
+
+        sortedscores: Dict[int, Dict[int, Dict[str, Union[Score, Achievement]]]] = {}
+
+        for score in scores:
+            if score.id not in sortedscores:
+                sortedscores[score.id] = {}
+            if score.chart not in sortedscores[score.id]:
+                sortedscores[score.id][score.chart] = {}
+            sortedscores[score.id][score.chart]['score'] = score
+
+        for oldscore in old_scores:
+            songid = int(oldscore.id / 100)
+            chart = int(oldscore.id % 100)
+            if songid not in sortedscores:
+                sortedscores[songid] = {}
+            if chart not in sortedscores[songid]:
+                sortedscores[songid][chart] = {}
+            sortedscores[songid][chart]['oldscore'] = oldscore
+
+        game = Node.void('game')
+        for song in sortedscores:
+            music = Node.void('music')
+            game.add_child(music)
+            music.set_attribute('reclink', str(song))
+
+            for chart in sortedscores[song]:
+                try:
+                    gamechart = self.db_to_game_chart(chart)
+                except KeyError:
+                    # Don't support this chart in this game
+                    continue
+                scoredict = sortedscores[song][chart]
+
+                if 'score' in scoredict:
+                    # We played the normal version of this song
+                    gamerank = self.db_to_game_rank(scoredict['score'].data.get_int('rank'))
+                    combo_type = self.db_to_game_halo(scoredict['score'].data.get_int('halo'))
+                    points = scoredict['score'].points  # type: ignore
+                    plays = scoredict['score'].plays  # type: ignore
+                else:
+                    # We only played 2nd mix version of this song
+                    gamerank = 0
+                    combo_type = self.GAME_HALO_NONE
+                    points = 0
+                    plays = 0
+
+                if 'oldscore' in scoredict:
+                    # We played the 2nd mix version of this song
+                    oldpoints = scoredict['oldscore'].data.get_int('points')
+                    oldrank = scoredict['oldscore'].data.get_int('rank')
+                    oldplays = scoredict['oldscore'].data.get_int('plays')
+                else:
+                    oldpoints = 0
+                    oldrank = 0
+                    oldplays = 0
+
+                typenode = Node.void('type')
+                music.add_child(typenode)
+                typenode.set_attribute('diff', str(gamechart))
+
+                typenode.add_child(Node.u32('score', points))
+                typenode.add_child(Node.u16('count', plays))
+                typenode.add_child(Node.u8('rank', gamerank))
+                typenode.add_child(Node.u8('combo_type', combo_type))
+                typenode.add_child(Node.u32('score_2nd', oldpoints))
+                typenode.add_child(Node.u8('rank_2nd', oldrank))
+                typenode.add_child(Node.u16('cnt_2nd', oldplays))
+
+        return game
+
+    def handle_game_save_m_request(self, request: Node) -> Node:
+        refid = request.attribute('refid')
+        songid = int(request.attribute('mid'))
+        chart = self.game_to_db_chart(int(request.attribute('mtype')))
+        userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+
+        # Calculate statistics
+        data = request.child('data')
+        playmode = int(data.attribute('playmode'))
+        if playmode == self.GAME_PLAY_MODE_2NDMIX:
+            if userid is not None:
+                # These are special cased and treated like achievements
+                points = int(data.attribute('score_2nd'))
+                combo = int(data.attribute('combo_2nd'))
+                rank = int(data.attribute('rank_2nd'))
+
+                # Grab the old 2nd mix score
+                existingscore = self.data.local.user.get_achievement(
+                    self.game,
+                    self.music_version,
+                    userid,
+                    (songid * 100) + chart,
+                    '2ndmix',
+                )
+
+                if existingscore is not None:
+                    highscore = points > existingscore.get_int('points')
+
+                    plays = existingscore.get_int('plays', 0) + 1
+                    points = max(points, existingscore.get_int('points'))
+                    if not highscore:
+                        combo = existingscore.get_int('combo', combo)
+                        rank = existingscore.get_int('rank', rank)
+                else:
+                    plays = 1
+
+                self.data.local.user.put_achievement(
+                    self.game,
+                    self.music_version,
+                    userid,
+                    (songid * 100) + chart,
+                    '2ndmix',
+                    {
+                        'points': points,
+                        'combo': combo,
+                        'rank': rank,
+                        'plays': plays,
+                    },
+                )
+        else:
+            points = int(data.attribute('score'))
+            combo = int(data.attribute('combo'))
+            rank = self.game_to_db_rank(int(data.attribute('rank')))
+            if int(data.attribute('full')) == 0:
+                halo = self.HALO_NONE
+            elif int(data.attribute('perf')) == 0:
+                halo = self.HALO_GREAT_FULL_COMBO
+            elif points < 1000000:
+                halo = self.HALO_PERFECT_FULL_COMBO
+            else:
+                halo = self.HALO_MARVELOUS_FULL_COMBO
+            trace = request.child_value('trace')
+
+            # Save the score, regardless of whether we have a refid. If we save
+            # an anonymous score, it only goes into the DB to count against the
+            # number of plays for that song/chart.
+            self.update_score(
+                userid,
+                songid,
+                chart,
+                points,
+                rank,
+                halo,
+                combo,
+                trace,
+            )
+
+        # No response needed
+        game = Node.void('game')
+        return game
+
+    def format_profile(self, userid: UserID, profile: ValidatedDict) -> Node:
+        root = Node.void('game')
+
+        # Look up play stats we bridge to every mix
+        play_stats = self.get_play_statistics(userid)
+
+        # Basic game settings
+        root.add_child(Node.string('seq', ''))
+        root.add_child(Node.u32('code', profile.get_int('extid')))
+        root.add_child(Node.string('name', profile.get_str('name')))
+        root.add_child(Node.u8('area', profile.get_int('area', 51)))
+        root.add_child(Node.u32('cnt_s', play_stats.get_int('single_plays')))
+        root.add_child(Node.u32('cnt_d', play_stats.get_int('double_plays')))
+        root.add_child(Node.u32('cnt_b', play_stats.get_int('battle_plays')))  # This could be wrong, its a guess
+        root.add_child(Node.u32('cnt_m0', play_stats.get_int('cnt_m0')))
+        root.add_child(Node.u32('cnt_m1', play_stats.get_int('cnt_m1')))
+        root.add_child(Node.u32('cnt_m2', play_stats.get_int('cnt_m2')))
+        root.add_child(Node.u32('cnt_m3', play_stats.get_int('cnt_m3')))
+        root.add_child(Node.u32('cnt_m4', play_stats.get_int('cnt_m4')))
+        root.add_child(Node.u32('cnt_m5', play_stats.get_int('cnt_m5')))
+        root.add_child(Node.u32('exp', play_stats.get_int('exp')))
+        root.add_child(Node.u32('exp_o', profile.get_int('exp_o')))
+        root.add_child(Node.u32('star', profile.get_int('star')))
+        root.add_child(Node.u32('star_c', profile.get_int('star_c')))
+        root.add_child(Node.u8('combo', profile.get_int('combo', 0)))
+        root.add_child(Node.u8('timing_diff', profile.get_int('early_late', 0)))
+
+        # Character stuff
+        chara = Node.void('chara')
+        root.add_child(chara)
+        chara.set_attribute('my', str(profile.get_int('chara', 30)))
+        root.add_child(Node.u16_array('chara_opt', profile.get_int_array('chara_opt', 96, [208] * 96)))
+
+        # Drill rankings
+        if 'title_gr' in profile:
+            title_gr = Node.void('title_gr')
+            root.add_child(title_gr)
+            title_grdict = profile.get_dict('title_gr')
+            if 't' in title_grdict:
+                title_gr.set_attribute('t', str(title_grdict.get_int('t')))
+            if 's' in title_grdict:
+                title_gr.set_attribute('s', str(title_grdict.get_int('s')))
+            if 'd' in title_grdict:
+                title_gr.set_attribute('d', str(title_grdict.get_int('d')))
+
+        # Calorie mode
+        if 'weight' in profile:
+            workouts = self.data.local.user.get_time_based_achievements(
+                self.game,
+                self.version,
+                userid,
+                achievementtype='workout',
+                since=Time.now() - Time.SECONDS_IN_DAY,
+            )
+            total = sum([w.data.get_int('calories') for w in workouts])
+            workout = Node.void('workout')
+            root.add_child(workout)
+            workout.set_attribute('weight', str(profile.get_int('weight')))
+            workout.set_attribute('day', str(total))
+            workout.set_attribute('disp', '1')
+
+        # Daily play counts
+        last_play_date = play_stats.get_int_array('last_play_date', 3)
+        today_play_date = Time.todays_date()
+        if (
+            last_play_date[0] == today_play_date[0] and
+            last_play_date[1] == today_play_date[1] and
+            last_play_date[2] == today_play_date[2]
+        ):
+            today_count = play_stats.get_int('today_plays', 0)
+        else:
+            today_count = 0
+        daycount = Node.void('daycount')
+        root.add_child(daycount)
+        daycount.set_attribute('playcount', str(today_count))
+
+        # Daily combo stuff, unknown how this works
+        dailycombo = Node.void('dailycombo')
+        root.add_child(dailycombo)
+        dailycombo.set_attribute('daily_combo', str(0))
+        dailycombo.set_attribute('daily_combo_lv', str(0))
+
+        # Last cursor settings
+        last = Node.void('last')
+        root.add_child(last)
+        lastdict = profile.get_dict('last')
+        last.set_attribute('rival1', str(lastdict.get_int('rival1', -1)))
+        last.set_attribute('rival2', str(lastdict.get_int('rival2', -1)))
+        last.set_attribute('rival3', str(lastdict.get_int('rival3', -1)))
+        last.set_attribute('fri', str(lastdict.get_int('rival1', -1)))  # This literally goes to the same memory in X3
+        last.set_attribute('style', str(lastdict.get_int('style')))
+        last.set_attribute('mode', str(lastdict.get_int('mode')))
+        last.set_attribute('cate', str(lastdict.get_int('cate')))
+        last.set_attribute('sort', str(lastdict.get_int('sort')))
+        last.set_attribute('mid', str(lastdict.get_int('mid')))
+        last.set_attribute('mtype', str(lastdict.get_int('mtype')))
+        last.set_attribute('cid', str(lastdict.get_int('cid')))
+        last.set_attribute('ctype', str(lastdict.get_int('ctype')))
+        last.set_attribute('sid', str(lastdict.get_int('sid')))
+
+        # Result stars
+        result_star = Node.void('result_star')
+        root.add_child(result_star)
+        result_stars = profile.get_int_array('result_stars', 9)
+        for i in range(9):
+            result_star.set_attribute('slot{}'.format(i + 1), str(result_stars[i]))
+
+        # Target stuff
+        target = Node.void('target')
+        root.add_child(target)
+        target.set_attribute('flag', str(profile.get_int('target_flag')))
+        target.set_attribute('setnum', str(profile.get_int('target_setnum')))
+
+        # Groove gauge level-ups
+        gr_s = Node.void('gr_s')
+        root.add_child(gr_s)
+        index = 1
+        for entry in profile.get_int_array('gr_s', 5):
+            gr_s.set_attribute('gr{}'.format(index), str(entry))
+            index = index + 1
+
+        gr_d = Node.void('gr_d')
+        root.add_child(gr_d)
+        index = 1
+        for entry in profile.get_int_array('gr_d', 5):
+            gr_d.set_attribute('gr{}'.format(index), str(entry))
+            index = index + 1
+
+        # Options in menus
+        root.add_child(Node.s16_array('opt', profile.get_int_array('opt', 16)))
+        root.add_child(Node.s16_array('opt_ex', profile.get_int_array('opt_ex', 16)))
+
+        # Unlock flags
+        root.add_child(Node.u8_array('flag', profile.get_int_array('flag', 256, [1] * 256)))
+
+        # Ranking display?
+        root.add_child(Node.u16_array('rank', profile.get_int_array('rank', 100)))
+
+        # Rivals
+        links = self.data.local.user.get_links(self.game, self.version, userid)
+        for link in links:
+            if link.type[:7] != 'friend_':
+                continue
+
+            pos = int(link.type[7:])
+            friend = self.get_profile(link.other_userid)
+            play_stats = self.get_play_statistics(link.other_userid)
+            if friend is not None:
+                friendnode = Node.void('friend')
+                root.add_child(friendnode)
+                friendnode.set_attribute('pos', str(pos))
+                friendnode.set_attribute('vs', '0')
+                friendnode.set_attribute('up', '0')
+                friendnode.add_child(Node.u32('code', friend.get_int('extid')))
+                friendnode.add_child(Node.string('name', friend.get_str('name')))
+                friendnode.add_child(Node.u8('area', friend.get_int('area', 51)))
+                friendnode.add_child(Node.u32('exp', play_stats.get_int('exp')))
+                friendnode.add_child(Node.u32('star', friend.get_int('star')))
+
+                # Drill rankings
+                if 'title' in friend:
+                    title = Node.void('title')
+                    friendnode.add_child(title)
+                    titledict = friend.get_dict('title')
+                    if 't' in titledict:
+                        title.set_attribute('t', str(titledict.get_int('t')))
+                    if 's' in titledict:
+                        title.set_attribute('s', str(titledict.get_int('s')))
+                    if 'd' in titledict:
+                        title.set_attribute('d', str(titledict.get_int('d')))
+
+                if 'title_gr' in friend:
+                    title_gr = Node.void('title_gr')
+                    friendnode.add_child(title_gr)
+                    title_grdict = friend.get_dict('title_gr')
+                    if 't' in title_grdict:
+                        title_gr.set_attribute('t', str(title_grdict.get_int('t')))
+                    if 's' in title_grdict:
+                        title_gr.set_attribute('s', str(title_grdict.get_int('s')))
+                    if 'd' in title_grdict:
+                        title_gr.set_attribute('d', str(title_grdict.get_int('d')))
+
+                # Groove gauge level-ups
+                gr_s = Node.void('gr_s')
+                friendnode.add_child(gr_s)
+                index = 1
+                for entry in friend.get_int_array('gr_s', 5):
+                    gr_s.set_attribute('gr{}'.format(index), str(entry))
+                    index = index + 1
+
+                gr_d = Node.void('gr_d')
+                friendnode.add_child(gr_d)
+                index = 1
+                for entry in friend.get_int_array('gr_d', 5):
+                    gr_d.set_attribute('gr{}'.format(index), str(entry))
+                    index = index + 1
+
+        # Play area
+        areas = profile.get_int_array('play_area', 55)
+        play_area = Node.void('play_area')
+        root.add_child(play_area)
+        for i in range(len(areas)):
+            play_area.set_attribute('play_cnt{}'.format(i), str(areas[i]))
+
+        return root
+
+    def unformat_profile(self, userid: UserID, request: Node, oldprofile: ValidatedDict) -> ValidatedDict:
+        newprofile = copy.deepcopy(oldprofile)
+        play_stats = self.get_play_statistics(userid)
+
+        # Grab last node and accessories so we can make decisions based on type
+        last = request.child('last')
+        lastdict = newprofile.get_dict('last')
+        mode = int(last.attribute('mode'))
+        style = int(last.attribute('style'))
+        is_dp = style == self.GAME_STYLE_DOUBLE
+
+        # Drill rankings
+        title = request.child('title')
+        title_gr = request.child('title_gr')
+        titledict = newprofile.get_dict('title')
+        title_grdict = newprofile.get_dict('title_gr')
+
+        # Groove radar level ups
+        gr = request.child('gr')
+
+        # Set the correct values depending on if we're single or double play
+        if is_dp:
+            play_stats.increment_int('double_plays')
+            if gr is not None:
+                newprofile.replace_int_array(
+                    'gr_d',
+                    5,
+                    [
+                        intish(gr.attribute('gr1')),
+                        intish(gr.attribute('gr2')),
+                        intish(gr.attribute('gr3')),
+                        intish(gr.attribute('gr4')),
+                        intish(gr.attribute('gr5')),
+                    ],
+                )
+            if title is not None:
+                titledict.replace_int('d', title.value)
+                newprofile.replace_dict('title', titledict)
+            if title_gr is not None:
+                title_grdict.replace_int('d', title.value)
+                newprofile.replace_dict('title_gr', title_grdict)
+        else:
+            play_stats.increment_int('single_plays')
+            if gr is not None:
+                newprofile.replace_int_array(
+                    'gr_s',
+                    5,
+                    [
+                        intish(gr.attribute('gr1')),
+                        intish(gr.attribute('gr2')),
+                        intish(gr.attribute('gr3')),
+                        intish(gr.attribute('gr4')),
+                        intish(gr.attribute('gr5')),
+                    ],
+                )
+            if title is not None:
+                titledict.replace_int('s', title.value)
+                newprofile.replace_dict('title', titledict)
+            if title_gr is not None:
+                title_grdict.replace_int('s', title.value)
+                newprofile.replace_dict('title_gr', title_grdict)
+        play_stats.increment_int('cnt_m{}'.format(mode))
+
+        # Result stars
+        result_star = request.child('result_star')
+        if result_star is not None:
+            newprofile.replace_int_array(
+                'result_stars',
+                9,
+                [
+                    intish(result_star.attribute('slot1')),
+                    intish(result_star.attribute('slot2')),
+                    intish(result_star.attribute('slot3')),
+                    intish(result_star.attribute('slot4')),
+                    intish(result_star.attribute('slot5')),
+                    intish(result_star.attribute('slot6')),
+                    intish(result_star.attribute('slot7')),
+                    intish(result_star.attribute('slot8')),
+                    intish(result_star.attribute('slot9')),
+                ],
+            )
+
+        # Target stuff
+        target = request.child('target')
+        if target is not None:
+            newprofile.replace_int('target_flag', intish(target.attribute('flag')))
+            newprofile.replace_int('target_setnum', intish(target.attribute('setnum')))
+
+        # Update last attributes
+        lastdict.replace_int('rival1', intish(last.attribute('rival1')))
+        lastdict.replace_int('rival2', intish(last.attribute('rival2')))
+        lastdict.replace_int('rival3', intish(last.attribute('rival3')))
+        lastdict.replace_int('style', intish(last.attribute('style')))
+        lastdict.replace_int('mode', intish(last.attribute('mode')))
+        lastdict.replace_int('cate', intish(last.attribute('cate')))
+        lastdict.replace_int('sort', intish(last.attribute('sort')))
+        lastdict.replace_int('mid', intish(last.attribute('mid')))
+        lastdict.replace_int('mtype', intish(last.attribute('mtype')))
+        lastdict.replace_int('cid', intish(last.attribute('cid')))
+        lastdict.replace_int('ctype', intish(last.attribute('ctype')))
+        lastdict.replace_int('sid', intish(last.attribute('sid')))
+        newprofile.replace_dict('last', lastdict)
+
+        # Grab character options
+        chara = request.child('chara')
+        if chara is not None:
+            newprofile.replace_int('chara', intish(chara.attribute('my')))
+        chara_opt = request.child('chara_opt')
+        if chara_opt is not None:
+            # A bug in old versions of AVS returns the wrong number for set
+            newprofile.replace_int_array('chara_opt', 96, chara_opt.value[:96])
+
+        # Options
+        opt = request.child('opt')
+        if opt is not None:
+            # A bug in old versions of AVS returns the wrong number for set
+            newprofile.replace_int_array('opt', 16, opt.value[:16])
+
+        # Experience and stars
+        exp = request.child_value('exp')
+        if exp is not None:
+            play_stats.replace_int('exp', play_stats.get_int('exp') + exp)
+        star = request.child_value('star')
+        if star is not None:
+            newprofile.replace_int('star', newprofile.get_int('star') + star)
+        star_c = request.child_value('star_c')
+        if star_c is not None:
+            newprofile.replace_int('star_c', newprofile.get_int('star_c') + exp)
+
+        # Update game flags
+        for child in request.children:
+            if child.name != 'flag':
+                continue
+            try:
+                value = int(child.attribute('data'))
+                offset = int(child.attribute('no'))
+            except ValueError:
+                continue
+
+            flags = newprofile.get_int_array('flag', 256, [1] * 256)
+            if offset < 0 or offset >= len(flags):
+                continue
+            flags[offset] = value
+            newprofile.replace_int_array('flag', 256, flags)
+
+        # Workout mode support
+        newweight = -1
+        oldweight = newprofile.get_int('weight')
+        for child in request.children:
+            if child.name != 'weight':
+                continue
+            newweight = child.value
+        if newweight < 0:
+            newweight = oldweight
+
+        # Either update or unset the weight depending on the game
+        if newweight == 0:
+            # Weight is unset or we declined to use this feature, remove from profile
+            if 'weight' in newprofile:
+                del newprofile['weight']
+        else:
+            # Weight has been set or previously retrieved, we should save calories
+            newprofile.replace_int('weight', newweight)
+            total = 0
+            for child in request.children:
+                if child.name != 'calory':
+                    continue
+                total += child.value
+            self.data.local.user.put_time_based_achievement(
+                self.game,
+                self.version,
+                userid,
+                0,
+                'workout',
+                {
+                    'calories': total,
+                    'weight': newweight,
+                },
+            )
+
+        # Look up old friends
+        oldfriends: List[Optional[UserID]] = [None] * 10
+        links = self.data.local.user.get_links(self.game, self.version, userid)
+        for link in links:
+            if link.type[:7] != 'friend_':
+                continue
+
+            pos = int(link.type[7:])
+            oldfriends[pos] = link.other_userid
+
+        # Save any rivals that were added/removed/changed
+        newfriends = oldfriends[:]
+        for child in request.children:
+            if child.name != 'friend':
+                continue
+
+            code = int(child.attribute('code'))
+            pos = int(child.attribute('pos'))
+
+            if pos >= 0 and pos < 10:
+                if code == 0:
+                    # We cleared this friend
+                    newfriends[pos] = None
+                else:
+                    # Try looking up the userid
+                    newfriends[pos] = self.data.remote.user.from_extid(self.game, self.version, code)
+
+        # Diff the set of links to determine updates
+        for i in range(10):
+            if newfriends[i] == oldfriends[i]:
+                continue
+
+            if newfriends[i] is None:
+                # Kill the rival in this location
+                self.data.local.user.destroy_link(
+                    self.game,
+                    self.version,
+                    userid,
+                    'friend_{}'.format(i),
+                    oldfriends[i],
+                )
+            elif oldfriends[i] is None:
+                # Add rival in this location
+                self.data.local.user.put_link(
+                    self.game,
+                    self.version,
+                    userid,
+                    'friend_{}'.format(i),
+                    newfriends[i],
+                    {},
+                )
+            else:
+                # Changed the rival here, kill the old one, add the new one
+                self.data.local.user.destroy_link(
+                    self.game,
+                    self.version,
+                    userid,
+                    'friend_{}'.format(i),
+                    oldfriends[i],
+                )
+                self.data.local.user.put_link(
+                    self.game,
+                    self.version,
+                    userid,
+                    'friend_{}'.format(i),
+                    newfriends[i],
+                    {},
+                )
+
+        # Play area counter
+        shop_area = int(request.attribute('shop_area'))
+        if shop_area >= 0 and shop_area < 55:
+            areas = newprofile.get_int_array('play_area', 55)
+            areas[shop_area] = areas[shop_area] + 1
+            newprofile.replace_int_array('play_area', 55, areas)
+
+        # Keep track of play statistics
+        self.update_play_statistics(userid, play_stats)
+
+        return newprofile

bemani/backend/ddr/factory.py

@@ -0,0 +1,108 @@
+from typing import Dict, Optional, Any
+
+from bemani.backend.base import Base, Factory
+from bemani.backend.ddr.stubs import (
+    DDRX,
+    DDRSuperNova2,
+    DDRSuperNova,
+    DDRExtreme,
+    DDR7thMix,
+    DDR6thMix,
+    DDR5thMix,
+    DDR4thMix,
+    DDR3rdMix,
+    DDR2ndMix,
+    DDR1stMix,
+)
+from bemani.backend.ddr.ddrx2 import DDRX2
+from bemani.backend.ddr.ddrx3 import DDRX3
+from bemani.backend.ddr.ddr2013 import DDR2013
+from bemani.backend.ddr.ddr2014 import DDR2014
+from bemani.backend.ddr.ddrace import DDRAce
+from bemani.backend.ddr.ddra20 import DDRA20
+from bemani.common import Model, VersionConstants
+from bemani.data import Data
+
+
+class DDRFactory(Factory):
+
+    MANAGED_CLASSES = [
+        DDR1stMix,
+        DDR2ndMix,
+        DDR4thMix,
+        DDR3rdMix,
+        DDR5thMix,
+        DDR6thMix,
+        DDR7thMix,
+        DDRExtreme,
+        DDRSuperNova,
+        DDRSuperNova2,
+        DDRX,
+        DDRX2,
+        DDRX3,
+        DDR2013,
+        DDR2014,
+        DDRAce,
+        DDRA20,
+    ]
+
+    @classmethod
+    def register_all(cls) -> None:
+        for game in ['HDX', 'JDX', 'KDX', 'MDX']:
+            Base.register(game, DDRFactory)
+
+    @classmethod
+    def create(cls, data: Data, config: Dict[str, Any], model: Model, parentmodel: Optional[Model]=None) -> Optional[Base]:
+
+        def version_from_date(date: int) -> Optional[int]:
+            if date < 2014051200:
+                return VersionConstants.DDR_2013
+            elif date >= 2014051200 and date < 2016033000:
+                return VersionConstants.DDR_2014
+            elif date >= 2016033000 and date < 2019042300:
+                return VersionConstants.DDR_ACE
+            elif date >= 2019042300:
+                return VersionConstants.DDR_A20
+            return None
+
+        if model.game == 'HDX':
+            return DDRX(data, config, model)
+        if model.game == 'JDX':
+            return DDRX2(data, config, model)
+        if model.game == 'KDX':
+            return DDRX3(data, config, model)
+        if model.game == 'MDX':
+            if model.version is None:
+                if parentmodel is None:
+                    return None
+
+                # We have no way to tell apart newer versions. However, we can make
+                # an educated guess if we happen to be summoned for old profile lookup.
+                if parentmodel.game not in ['HDX', 'JDX', 'KDX', 'MDX']:
+                    return None
+
+                parentversion = version_from_date(parentmodel.version)
+                if parentversion == VersionConstants.DDR_A20:
+                    return DDRAce(data, config, model)
+                if parentversion == VersionConstants.DDR_ACE:
+                    return DDR2014(data, config, model)
+                if parentversion == VersionConstants.DDR_2014:
+                    return DDR2013(data, config, model)
+                if parentversion == VersionConstants.DDR_2013:
+                    return DDRX3(data, config, model)
+
+                # Unknown older version
+                return None
+
+            version = version_from_date(model.version)
+            if version == VersionConstants.DDR_2013:
+                return DDR2013(data, config, model)
+            if version == VersionConstants.DDR_2014:
+                return DDR2014(data, config, model)
+            if version == VersionConstants.DDR_ACE:
+                return DDRAce(data, config, model)
+            if version == VersionConstants.DDR_A20:
+                return DDRA20(data, config, model)
+
+        # Unknown game version
+        return None

bemani/backend/ddr/stubs.py

@@ -0,0 +1,101 @@
+# vim: set fileencoding=utf-8
+from typing import Optional
+
+from bemani.backend.ddr.base import DDRBase
+from bemani.common import VersionConstants
+
+
+class DDRX(DDRBase):
+
+    name = 'DanceDanceRevolution X'
+    version = VersionConstants.DDR_X
+
+    def previous_version(self) -> Optional[DDRBase]:
+        return DDRSuperNova2(self.data, self.config, self.model)
+
+
+class DDRSuperNova2(DDRBase):
+
+    name = 'DanceDanceRevolution SuperNova 2'
+    version = VersionConstants.DDR_SUPERNOVA_2
+
+    def previous_version(self) -> Optional[DDRBase]:
+        return DDRSuperNova(self.data, self.config, self.model)
+
+
+class DDRSuperNova(DDRBase):
+
+    name = 'DanceDanceRevolution SuperNova'
+    version = VersionConstants.DDR_SUPERNOVA
+
+    def previous_version(self) -> Optional[DDRBase]:
+        return DDRExtreme(self.data, self.config, self.model)
+
+
+class DDRExtreme(DDRBase):
+
+    name = 'DanceDanceRevolution Extreme'
+    version = VersionConstants.DDR_EXTREME
+
+    def previous_version(self) -> Optional[DDRBase]:
+        return DDR7thMix(self.data, self.config, self.model)
+
+
+class DDR7thMix(DDRBase):
+
+    name = 'DanceDanceRevolution 7thMix'
+    version = VersionConstants.DDR_7THMIX
+
+    def previous_version(self) -> Optional[DDRBase]:
+        return DDR6thMix(self.data, self.config, self.model)
+
+
+class DDR6thMix(DDRBase):
+
+    name = 'DanceDanceRevolution 6thMix'
+    version = VersionConstants.DDR_6THMIX
+
+    def previous_version(self) -> Optional[DDRBase]:
+        return DDR5thMix(self.data, self.config, self.model)
+
+
+class DDR5thMix(DDRBase):
+
+    name = 'DanceDanceRevolution 5thMix'
+    version = VersionConstants.DDR_5THMIX
+
+    def previous_version(self) -> Optional[DDRBase]:
+        return DDR4thMix(self.data, self.config, self.model)
+
+
+class DDR4thMix(DDRBase):
+
+    name = 'DanceDanceRevolution 4thMix'
+    version = VersionConstants.DDR_4THMIX
+
+    def previous_version(self) -> Optional[DDRBase]:
+        return DDR3rdMix(self.data, self.config, self.model)
+
+
+class DDR3rdMix(DDRBase):
+
+    name = 'DanceDanceRevolution 3rdMix'
+    version = VersionConstants.DDR_3RDMIX
+
+    def previous_version(self) -> Optional[DDRBase]:
+        return DDR2ndMix(self.data, self.config, self.model)
+
+
+class DDR2ndMix(DDRBase):
+
+    name = 'DanceDanceRevolution 2ndMix'
+    version = VersionConstants.DDR_2NDMIX
+
+    def previous_version(self) -> Optional[DDRBase]:
+        return DDR1stMix(self.data, self.config, self.model)
+
+
+class DDR1stMix(DDRBase):
+
+    name = 'DanceDanceRevolution 1stMix'
+    version = VersionConstants.DDR_1STMIX

bemani/backend/dispatch.py

@@ -0,0 +1,174 @@
+import copy
+from typing import Optional, Dict, Any
+
+from bemani.backend.base import Model, Base, Status
+from bemani.protocol import Node
+from bemani.data import Data
+
+
+class UnrecognizedPCBIDException(Exception):
+    def __init__(self, pcbid: str, model: str, ip: str) -> None:
+        self.pcbid = pcbid
+        self.model = model
+        self.ip = ip
+
+
+class Dispatch:
+    """
+    Dispatch object responsible for taking a decoded tree of Node objects
+    from a game, looking up config, dispatching it to the correct game
+    class and then returning a response.
+    """
+
+    def __init__(self, config: Dict[str, Any], data: Data, verbose: bool) -> None:
+        """
+        Initialize the Dispatch object.
+
+        Parameters:
+            config - A dictionary of configuration used for various settigs.
+            data - A Data singleton for DB access.
+            verbose - Whether we get chatty to stdout or not.
+        """
+        self.__verbose = verbose
+        self.__data = data
+        self.__config = config
+
+    def log(self, msg: str, *args: Any, **kwargs: Any) -> None:
+        """
+        Given a message, format it and print it.
+
+        Note that this only prints to stdout if we were initialized with
+        verbose = True.
+
+        Parameters:
+            msg - A formatstring that should be formatted with any
+                  optional arguments or keyword arguments.
+        """
+        if self.__verbose:
+            print(msg.format(*args, **kwargs))
+
+    def handle(self, tree: Node) -> Optional[Node]:
+        """
+        Given a packet from a game, handle it and return a response.
+
+        Parameters:
+            tree - A Node representing the root of a tree. Expected to
+                   come from an external game.
+
+        Returns:
+            A Node representing the root of a response tree, or None if
+            we had a problem parsing or generating a response.
+        """
+        self.log("Received request:\n{}", tree)
+
+        if tree.name != 'call':
+            # Invalid request
+            self.log("Invalid root node {}", tree.name)
+            return None
+
+        if len(tree.children) != 1:
+            # Invalid request
+            self.log("Invalid number of children for root node")
+            return None
+
+        modelstring = tree.attribute('model')
+        model = Model.from_modelstring(modelstring)
+        pcbid = tree.attribute('srcid')
+
+        # If we are enforcing, bail out if we don't recognize thie ID
+        pcb = self.__data.local.machine.get_machine(pcbid)
+        if self.__config['server']['enforce_pcbid'] and pcb is None:
+            self.log("Unrecognized PCBID {}", pcbid)
+            raise UnrecognizedPCBIDException(pcbid, modelstring, self.__config['client']['address'])
+
+        # If we don't have a Machine, but we aren't enforcing, we must create it
+        if pcb is None:
+            pcb = self.__data.local.machine.create_machine(pcbid)
+
+        request = tree.children[0]
+
+        config = copy.copy(self.__config)
+        config['machine'] = {
+            'pcbid': pcbid,
+            'arcade': pcb.arcade,
+        }
+
+        # If the machine we looked up is in an arcade, override the global
+        # paseli settings with the arcade paseli settings.
+        if pcb.arcade is not None:
+            arcade = self.__data.local.machine.get_arcade(pcb.arcade)
+            if arcade is not None:
+                config['paseli']['enabled'] = arcade.data.get_bool('paseli_enabled')
+                config['paseli']['infinite'] = arcade.data.get_bool('paseli_infinite')
+                if arcade.data.get_bool('mask_services_url'):
+                    # Mask the address, no matter what the server settings are
+                    config['server']['uri'] = None
+        # If we don't have a server URI, we should add the default
+        if 'uri' not in config['server']:
+            config['server']['uri'] = None
+
+        game = Base.create(self.__data, config, model)
+        method = request.attribute('method')
+        response = None
+
+        # If we are enforcing, make sure the PCBID isn't specified to be
+        # game-specific
+        if self.__config['server']['enforce_pcbid'] and pcb.game is not None:
+            if pcb.game != game.game:
+                self.log("PCBID {} assigned to game {}, but connected from game {}", pcbid, pcb.game, game.game)
+                raise UnrecognizedPCBIDException(pcbid, modelstring, self.__config['client']['address'])
+            if pcb.version is not None:
+                if pcb.version > 0 and pcb.version != game.version:
+                    self.log(
+                        "PCBID {} assigned to game {} version {}, but connected from game {} version {}",
+                        pcbid,
+                        pcb.game,
+                        pcb.version,
+                        game.game,
+                        game.version,
+                    )
+                    raise UnrecognizedPCBIDException(pcbid, modelstring, self.__config['client']['address'])
+                if pcb.version < 0 and (-pcb.version) < game.version:
+                    self.log(
+                        "PCBID {} assigned to game {} maximum version {}, but connected from game {} version {}",
+                        pcbid,
+                        pcb.game,
+                        -pcb.version,
+                        game.game,
+                        game.version,
+                    )
+                    raise UnrecognizedPCBIDException(pcbid, modelstring, self.__config['client']['address'])
+
+        # First, try to handle with specific service/method function
+        try:
+            handler = getattr(game, 'handle_{}_{}_request'.format(request.name, method))
+        except AttributeError:
+            handler = None
+        if handler is not None:
+            response = handler(request)
+
+        if response is None:
+            # Now, try to pass it off to a generic service handler
+            try:
+                handler = getattr(game, 'handle_{}_request'.format(request.name))
+            except AttributeError:
+                handler = None
+            if handler is not None:
+                response = handler(request)
+
+        if response is None:
+            # Unrecognized handler
+            self.log("Unrecognized service {} method {}".format(request.name, method))
+            return None
+
+        # Make sure we have a status value if one wasn't provided
+        if 'status' not in response.attributes:
+            response.set_attribute('status', str(Status.SUCCESS))
+
+        root = Node.void('response')
+        root.add_child(response)
+        root.set_attribute('dstid', pcbid)
+
+        self.log("Sending response:\n{}", root)
+
+        return root

bemani/backend/ess/__init__.py

@@ -0,0 +1 @@
+from bemani.backend.ess.eventlog import EventLogHandler

bemani/backend/ess/eventlog.py

@@ -0,0 +1,23 @@
+import random
+
+from bemani.backend.base import Base
+from bemani.protocol import Node
+
+
+class EventLogHandler(Base):
+    """
+    A mixin that can be used to provide ESS eventlog handling.
+    """
+
+    def handle_eventlog_write_request(self, request: Node) -> Node:
+        # Just turn off further logging
+        gamesession = request.child_value('data/gamesession')
+        if gamesession < 0:
+            gamesession = random.randint(1, 1000000)
+
+        root = Node.void('eventlog')
+        root.add_child(Node.s64('gamesession', gamesession))
+        root.add_child(Node.s32('logsendflg', 0))
+        root.add_child(Node.s32('logerrlevel', 0))
+        root.add_child(Node.s32('evtidnosendflg', 0))
+        return root

bemani/backend/iidx/__init__.py

@@ -0,0 +1,2 @@
+from bemani.backend.iidx.factory import IIDXFactory
+from bemani.backend.iidx.base import IIDXBase

bemani/backend/iidx/base.py

@@ -0,0 +1,818 @@
+# vim: set fileencoding=utf-8
+import struct
+from typing import Optional, Dict, Any, List, Tuple
+
+from bemani.backend.base import Base
+from bemani.backend.core import CoreHandler, CardManagerHandler, PASELIHandler
+from bemani.common import ValidatedDict, Model, GameConstants, DBConstants, Parallel
+from bemani.data import Data, Score, Machine, UserID
+from bemani.protocol import Node
+
+
+class IIDXBase(CoreHandler, CardManagerHandler, PASELIHandler, Base):
+    """
+    Base game class for all Beatmania IIDX versions. Handles common functionality for
+    getting profiles based on refid, creating new profiles, looking up and saving
+    scores.
+    """
+
+    game = GameConstants.IIDX
+
+    paseli_padding = 15
+
+    CLEAR_TYPE_SINGLE = 1
+    CLEAR_TYPE_DOUBLE = 2
+
+    CLEAR_STATUS_NO_PLAY = DBConstants.IIDX_CLEAR_STATUS_NO_PLAY
+    CLEAR_STATUS_FAILED = DBConstants.IIDX_CLEAR_STATUS_FAILED
+    CLEAR_STATUS_ASSIST_CLEAR = DBConstants.IIDX_CLEAR_STATUS_ASSIST_CLEAR
+    CLEAR_STATUS_EASY_CLEAR = DBConstants.IIDX_CLEAR_STATUS_EASY_CLEAR
+    CLEAR_STATUS_CLEAR = DBConstants.IIDX_CLEAR_STATUS_CLEAR
+    CLEAR_STATUS_HARD_CLEAR = DBConstants.IIDX_CLEAR_STATUS_HARD_CLEAR
+    CLEAR_STATUS_EX_HARD_CLEAR = DBConstants.IIDX_CLEAR_STATUS_EX_HARD_CLEAR
+    CLEAR_STATUS_FULL_COMBO = DBConstants.IIDX_CLEAR_STATUS_FULL_COMBO
+
+    CHART_TYPE_N7 = 0
+    CHART_TYPE_H7 = 1
+    CHART_TYPE_A7 = 2
+    CHART_TYPE_N14 = 3
+    CHART_TYPE_H14 = 4
+    CHART_TYPE_A14 = 5
+    # Beginner charts only save status
+    CHART_TYPE_B7 = 6
+
+    DAN_RANK_7_KYU = DBConstants.IIDX_DAN_RANK_7_KYU
+    DAN_RANK_6_KYU = DBConstants.IIDX_DAN_RANK_6_KYU
+    DAN_RANK_5_KYU = DBConstants.IIDX_DAN_RANK_5_KYU
+    DAN_RANK_4_KYU = DBConstants.IIDX_DAN_RANK_4_KYU
+    DAN_RANK_3_KYU = DBConstants.IIDX_DAN_RANK_3_KYU
+    DAN_RANK_2_KYU = DBConstants.IIDX_DAN_RANK_2_KYU
+    DAN_RANK_1_KYU = DBConstants.IIDX_DAN_RANK_1_KYU
+    DAN_RANK_1_DAN = DBConstants.IIDX_DAN_RANK_1_DAN
+    DAN_RANK_2_DAN = DBConstants.IIDX_DAN_RANK_2_DAN
+    DAN_RANK_3_DAN = DBConstants.IIDX_DAN_RANK_3_DAN
+    DAN_RANK_4_DAN = DBConstants.IIDX_DAN_RANK_4_DAN
+    DAN_RANK_5_DAN = DBConstants.IIDX_DAN_RANK_5_DAN
+    DAN_RANK_6_DAN = DBConstants.IIDX_DAN_RANK_6_DAN
+    DAN_RANK_7_DAN = DBConstants.IIDX_DAN_RANK_7_DAN
+    DAN_RANK_8_DAN = DBConstants.IIDX_DAN_RANK_8_DAN
+    DAN_RANK_9_DAN = DBConstants.IIDX_DAN_RANK_9_DAN
+    DAN_RANK_10_DAN = DBConstants.IIDX_DAN_RANK_10_DAN
+    DAN_RANK_CHUDEN = DBConstants.IIDX_DAN_RANK_CHUDEN
+    DAN_RANK_KAIDEN = DBConstants.IIDX_DAN_RANK_KAIDEN
+
+    DAN_RANKING_SINGLE = 'sgrade'
+    DAN_RANKING_DOUBLE = 'dgrade'
+
+    GHOST_TYPE_NONE = 0
+    GHOST_TYPE_RIVAL = 100
+    GHOST_TYPE_GLOBAL_TOP = 200
+    GHOST_TYPE_GLOBAL_AVERAGE = 300
+    GHOST_TYPE_LOCAL_TOP = 400
+    GHOST_TYPE_LOCAL_AVERAGE = 500
+    GHOST_TYPE_DAN_TOP = 600
+    GHOST_TYPE_DAN_AVERAGE = 700
+    GHOST_TYPE_RIVAL_TOP = 800
+    GHOST_TYPE_RIVAL_AVERAGE = 900
+
+    def __init__(self, data: Data, config: Dict[str, Any], model: Model) -> None:
+        super().__init__(data, config, model)
+        if model.rev == 'X':
+            self.omnimix = True
+        else:
+            self.omnimix = False
+
+    @property
+    def music_version(self) -> int:
+        if self.omnimix:
+            return DBConstants.OMNIMIX_VERSION_BUMP + self.version
+        return self.version
+
+    def previous_version(self) -> Optional['IIDXBase']:
+        """
+        Returns the previous version of the game, based on this game. Should
+        be overridden.
+        """
+        return None
+
+    def extra_services(self) -> List[str]:
+        """
+        Return the local2 service so that Copula and above will send certain packets.
+        """
+        return [
+            'local2',
+        ]
+
+    def format_profile(self, userid: UserID, profile: ValidatedDict) -> Node:
+        """
+        Base handler for a profile. Given a userid and a profile dictionary,
+        return a Node representing a profile. Should be overridden.
+        """
+        return Node.void('pc')
+
+    def unformat_profile(self, userid: UserID, request: Node, oldprofile: ValidatedDict) -> ValidatedDict:
+        """
+        Base handler for profile parsing. Given a request and an old profile,
+        return a new profile that's been updated with the contents of the request.
+        Should be overridden.
+        """
+        return oldprofile
+
+    def get_profile_by_refid(self, refid: Optional[str]) -> Optional[Node]:
+        """
+        Given a RefID, return a formatted profile node. Basically every game
+        needs a profile lookup, even if it handles where that happens in
+        a different request. This is provided for code deduplication.
+        """
+        if refid is None:
+            return None
+
+        userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        if userid is None:
+            # User doesn't exist but should at this point
+            return None
+
+        # Trying to import from current version
+        profile = self.get_profile(userid)
+        if profile is None:
+            return None
+        return self.format_profile(userid, profile)
+
+    def new_profile_by_refid(self, refid: Optional[str], name: Optional[str], pid: Optional[int]) -> ValidatedDict:
+        """
+        Given a RefID and an optional name, create a profile and then return
+        that newly created profile.
+        """
+        if refid is None:
+            return None
+
+        if name is None:
+            name = 'なし'
+        if pid is None:
+            pid = 51
+
+        userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        defaultprofile = ValidatedDict({
+            'name': name,
+            'pid': pid,
+            'settings': {
+                'flags': 223  # Default to turning on all optional folders
+            },
+        })
+        self.put_profile(userid, defaultprofile)
+        profile = self.get_profile(userid)
+        return profile
+
+    def put_profile_by_extid(self, extid: Optional[int], request: Node) -> None:
+        """
+        Given an ExtID and a request node, unformat the profile and save it.
+        """
+        userid = self.data.remote.user.from_extid(self.game, self.version, extid)
+        if userid is None:
+            return
+
+        oldprofile = self.get_profile(userid)
+        newprofile = self.unformat_profile(userid, request, oldprofile)
+        if newprofile is not None:
+            self.put_profile(userid, newprofile)
+
+    def get_machine_by_id(self, shop_id: int) -> Optional[Machine]:
+        pcbid = self.data.local.machine.from_machine_id(shop_id)
+        if pcbid is not None:
+            return self.data.local.machine.get_machine(pcbid)
+        else:
+            return None
+
+    def machine_joined_arcade(self) -> bool:
+        machine = self.data.local.machine.get_machine(self.config['machine']['pcbid'])
+        return machine.arcade is not None
+
+    def get_clear_rates(
+        self,
+        songid: Optional[int]=None,
+        songchart: Optional[int]=None,
+    ) -> Dict[int, Dict[int, Dict[str, int]]]:
+        """
+        Returns a dictionary similar to the following:
+
+        {
+            musicid: {
+                chart: {
+                    total: total plays,
+                    clears: total clears,
+                    fcs: total full combos,
+                },
+            },
+        }
+        """
+        all_attempts, remote_attempts = Parallel.execute([
+            lambda: self.data.local.music.get_all_attempts(
+                game=self.game,
+                version=self.music_version,
+                songid=songid,
+                songchart=songchart,
+            ),
+            lambda: self.data.remote.music.get_clear_rates(
+                game=self.game,
+                version=self.music_version,
+                songid=songid,
+                songchart=songchart,
+            ),
+        ])
+
+        attempts: Dict[int, Dict[int, Dict[str, int]]] = {}
+        for (_, attempt) in all_attempts:
+            if attempt.data.get_int('clear_status') == self.CLEAR_STATUS_NO_PLAY:
+                # This attempt was outside of the clear infra, so don't bother with it.
+                continue
+
+            # Terrible temporary structure is terrible.
+            if attempt.id not in attempts:
+                attempts[attempt.id] = {}
+            if attempt.chart not in attempts[attempt.id]:
+                attempts[attempt.id][attempt.chart] = {
+                    'total': 0,
+                    'clears': 0,
+                    'fcs': 0,
+                }
+
+            # We saw an attempt, keep the total attempts in sync.
+            attempts[attempt.id][attempt.chart]['total'] = attempts[attempt.id][attempt.chart]['total'] + 1
+
+            if attempt.data.get_int('clear_status', self.CLEAR_STATUS_FAILED) == self.CLEAR_STATUS_FAILED:
+                # This attempt was a failure, so don't count it against clears of full combos
+                continue
+
+            # It was at least a clear
+            attempts[attempt.id][attempt.chart]['clears'] = attempts[attempt.id][attempt.chart]['clears'] + 1
+
+            if attempt.data.get_int('clear_status') == self.CLEAR_STATUS_FULL_COMBO:
+                # This was a full combo clear, so it also counts here
+                attempts[attempt.id][attempt.chart]['fcs'] = attempts[attempt.id][attempt.chart]['fcs'] + 1
+
+        # Merge in remote attempts
+        for songid in remote_attempts:
+            if songid not in attempts:
+                attempts[songid] = {}
+
+            for songchart in remote_attempts[songid]:
+                if songchart not in attempts[songid]:
+                    attempts[songid][songchart] = {
+                        'total': 0,
+                        'clears': 0,
+                        'fcs': 0,
+                    }
+
+                attempts[songid][songchart]['total'] += remote_attempts[songid][songchart]['plays']
+                attempts[songid][songchart]['clears'] += remote_attempts[songid][songchart]['clears']
+                attempts[songid][songchart]['fcs'] += remote_attempts[songid][songchart]['combos']
+
+        # If requesting a specific song/chart, make sure its in the dict
+        if songid is not None:
+            if songid not in attempts:
+                attempts[songid] = {}
+
+            if songchart is not None:
+                if songchart not in attempts[songid]:
+                    attempts[songid][songchart] = {
+                        'total': 0,
+                        'clears': 0,
+                        'fcs': 0,
+                    }
+
+        return attempts
+
+    def update_score(
+        self,
+        userid: Optional[UserID],
+        songid: int,
+        chart: int,
+        clear_status: int,
+        pgreats: int,
+        greats: int,
+        miss_count: int,
+        ghost: Optional[bytes],
+        shop: Optional[int],
+    ) -> None:
+        """
+        Given various pieces of a score, update the user's high score and score
+        history in a controlled manner, so all games in IIDX series can expect
+        the same attributes in a score. Note that the medals passed here are
+        expected to be converted from game identifier to our internal identifier,
+        so that any game in the series may convert them back. In this way, a song
+        played on Pendual that exists in Tricoro will still have scores/medals
+        going back all versions.
+        """
+        # Range check medals
+        if clear_status not in [
+            self.CLEAR_STATUS_NO_PLAY,
+            self.CLEAR_STATUS_FAILED,
+            self.CLEAR_STATUS_ASSIST_CLEAR,
+            self.CLEAR_STATUS_EASY_CLEAR,
+            self.CLEAR_STATUS_CLEAR,
+            self.CLEAR_STATUS_HARD_CLEAR,
+            self.CLEAR_STATUS_EX_HARD_CLEAR,
+            self.CLEAR_STATUS_FULL_COMBO,
+        ]:
+            raise Exception("Invalid clear status value {}".format(clear_status))
+
+        # Calculate ex score
+        ex_score = (2 * pgreats) + greats
+
+        if userid is not None:
+            if ghost is None:
+                raise Exception("Expected a ghost for user score save!")
+            oldscore = self.data.local.music.get_score(
+                self.game,
+                self.music_version,
+                userid,
+                songid,
+                chart,
+            )
+        else:
+            # Storing an anonymous attempt
+            if ghost is not None:
+                raise Exception("Expected no ghost for anonymous score save!")
+            oldscore = None
+
+        # Score history is verbatum, instead of highest score
+        history = ValidatedDict({
+            'clear_status': clear_status,
+            'miss_count': miss_count,
+        })
+        old_ex_score = ex_score
+
+        if ghost is not None:
+            history['ghost'] = ghost
+
+        if oldscore is None:
+            # If it is a new score, create a new dictionary to add to
+            scoredata = ValidatedDict({
+                'clear_status': clear_status,
+                'miss_count': miss_count,
+                'pgreats': pgreats,
+                'greats': greats,
+            })
+            if ghost is not None:
+                scoredata['ghost'] = ghost
+            raised = True
+            highscore = True
+        else:
+            # Set the score to any new record achieved
+            raised = ex_score > oldscore.points
+            highscore = ex_score >= oldscore.points
+            ex_score = max(ex_score, oldscore.points)
+            scoredata = oldscore.data
+            scoredata.replace_int('clear_status', max(scoredata.get_int('clear_status'), clear_status))
+            if raised:
+                scoredata.replace_int('miss_count', miss_count)
+                scoredata.replace_int('pgreats', pgreats)
+                scoredata.replace_int('greats', greats)
+                if ghost is not None:
+                    scoredata.replace_bytes('ghost', ghost)
+
+        if shop is not None:
+            history.replace_int('shop', shop)
+            scoredata.replace_int('shop', shop)
+
+        # Look up where this score was earned
+        lid = self.get_machine_id()
+
+        if userid is not None:
+            # Write the new score back
+            self.data.local.music.put_score(
+                self.game,
+                self.music_version,
+                userid,
+                songid,
+                chart,
+                lid,
+                ex_score,
+                scoredata,
+                highscore,
+            )
+
+        # Save the history of this score too
+        self.data.local.music.put_attempt(
+            self.game,
+            self.music_version,
+            userid,
+            songid,
+            chart,
+            lid,
+            old_ex_score,
+            history,
+            raised,
+        )
+
+    def update_rank(
+        self,
+        userid: UserID,
+        dantype: str,
+        rank: int,
+        percent: int,
+        cleared: bool,
+        stages_cleared: int,
+    ) -> None:
+        # Range check type
+        if dantype not in [
+            self.DAN_RANKING_SINGLE,
+            self.DAN_RANKING_DOUBLE,
+        ]:
+            raise Exception("Invalid dan rank type value {}".format(dantype))
+
+        # Range check rank
+        if rank not in [
+            self.DAN_RANK_7_KYU,
+            self.DAN_RANK_6_KYU,
+            self.DAN_RANK_5_KYU,
+            self.DAN_RANK_4_KYU,
+            self.DAN_RANK_3_KYU,
+            self.DAN_RANK_2_KYU,
+            self.DAN_RANK_1_KYU,
+            self.DAN_RANK_1_DAN,
+            self.DAN_RANK_2_DAN,
+            self.DAN_RANK_3_DAN,
+            self.DAN_RANK_4_DAN,
+            self.DAN_RANK_5_DAN,
+            self.DAN_RANK_6_DAN,
+            self.DAN_RANK_7_DAN,
+            self.DAN_RANK_8_DAN,
+            self.DAN_RANK_9_DAN,
+            self.DAN_RANK_10_DAN,
+            self.DAN_RANK_CHUDEN,
+            self.DAN_RANK_KAIDEN,
+        ]:
+            raise Exception("Invalid dan rank {}".format(rank))
+
+        if cleared:
+            # Update profile if needed
+            profile = self.get_profile(userid)
+            if profile is None:
+                profile = ValidatedDict()
+
+            profile.replace_int(dantype, max(rank, profile.get_int(dantype, -1)))
+            self.put_profile(userid, profile)
+
+        # Update achievement to track pass rate
+        dan_score = self.data.local.user.get_achievement(
+            self.game,
+            self.version,
+            userid,
+            rank,
+            dantype,
+        )
+        if dan_score is None:
+            dan_score = ValidatedDict()
+        dan_score.replace_int('percent', max(percent, dan_score.get_int('percent')))
+        dan_score.replace_int('stages_cleared', max(stages_cleared, dan_score.get_int('stages_cleared')))
+        self.data.local.user.put_achievement(
+            self.game,
+            self.version,
+            userid,
+            rank,
+            dantype,
+            dan_score
+        )
+
+    def db_to_game_status(self, db_status: int) -> int:
+        """
+        Given a DB status, translate to a game clear status.
+        """
+        raise Exception('Implement in specific game class!')
+
+    def game_to_db_status(self, game_status: int) -> int:
+        """
+        Given a game clear status, translate to DB status.
+        """
+        raise Exception('Implement in specific game class!')
+
+    def make_score_struct(self, scores: List[Score], cltype: int, index: int) -> List[List[int]]:
+        scorestruct: Dict[int, List[int]] = {}
+
+        for score in scores:
+            musicid = score.id
+            chart = score.chart
+
+            # Filter to only singles/doubles charts
+            if cltype == self.CLEAR_TYPE_SINGLE:
+                if chart not in [
+                    self.CHART_TYPE_N7,
+                    self.CHART_TYPE_H7,
+                    self.CHART_TYPE_A7,
+                ]:
+                    continue
+                chartindex = {
+                    self.CHART_TYPE_N7: 0,
+                    self.CHART_TYPE_H7: 1,
+                    self.CHART_TYPE_A7: 2,
+                }[chart]
+            if cltype == self.CLEAR_TYPE_DOUBLE:
+                if chart not in [
+                    self.CHART_TYPE_N14,
+                    self.CHART_TYPE_H14,
+                    self.CHART_TYPE_A14,
+                ]:
+                    continue
+                chartindex = {
+                    self.CHART_TYPE_N14: 0,
+                    self.CHART_TYPE_H14: 1,
+                    self.CHART_TYPE_A14: 2,
+                }[chart]
+
+            if musicid not in scorestruct:
+                scorestruct[musicid] = [
+                    index,  # -1 is our scores, positive is rival index
+                    musicid,  # Music ID!
+                    0,  # Normal status,
+                    0,  # Hyper status,
+                    0,  # Another status,
+                    0,  # EX score normal,
+                    0,  # EX score hyper,
+                    0,  # EX score another,
+                    -1,  # Miss count normal,
+                    -1,  # Miss count hyper,
+                    -1,  # Miss count another,
+                ]
+
+            scorestruct[musicid][chartindex + 2] = self.db_to_game_status(score.data.get_int('clear_status'))
+            scorestruct[musicid][chartindex + 5] = score.points
+            scorestruct[musicid][chartindex + 8] = score.data.get_int('miss_count', -1)
+
+        return [scorestruct[s] for s in scorestruct]
+
+    def make_beginner_struct(self, scores: List[Score]) -> List[List[int]]:
+        scorelist: List[List[int]] = []
+
+        for score in scores:
+            musicid = score.id
+            chart = score.chart
+
+            # Filter to only beginner charts
+            if chart != self.CHART_TYPE_B7:
+                continue
+
+            scorelist.append([
+                musicid,
+                self.db_to_game_status(score.data.get_int('clear_status')),
+            ])
+
+        return scorelist
+
+    def delta_score(
+        self,
+        scores: List[Score],
+        ghost_length: int,
+    ) -> Tuple[Optional[int], Optional[bytes]]:
+        if len(scores) == 0:
+            return None, None
+
+        total_ghost = [0] * ghost_length
+        count = 0
+
+        # Sum up for each bucket
+        for score in scores:
+            ghost = score.data.get_bytes('ghost')
+            for i in range(len(ghost)):
+                total_ghost[i] = total_ghost[i] + ghost[i]
+            count = count + 1
+
+        # Calculate average for each bucket
+        total_ghost = [int(b / count) for b in total_ghost]
+
+        # Grab the ex score for this new ghost, being sure to reverse the scaling rate
+        new_ex_score = sum(total_ghost)
+
+        # Spread out into even buckets so we can compute deltas
+        reference_ghost = [int(new_ex_score / ghost_length)] * ghost_length
+
+        added_bucket = 0
+        try:
+            jump = max(1, int(ghost_length / (new_ex_score - sum(reference_ghost))))
+        except ZeroDivisionError:
+            jump = 1
+        while sum(reference_ghost) != new_ex_score:
+            reference_ghost[added_bucket] = reference_ghost[added_bucket] + 1
+            added_bucket = added_bucket + jump
+
+        # Calculate delta ghost
+        delta_ghost = [total_ghost[i] - reference_ghost[i] for i in range(ghost_length)]
+
+        # Return averages
+        return new_ex_score, struct.pack('b' * ghost_length, *delta_ghost)
+
+    def user_joined_arcade(self, machine: Machine, profile: Optional[ValidatedDict]) -> bool:
+        if profile is None:
+            return False
+
+        if 'shop_location' not in profile:
+            return False
+
+        machineid = profile.get_int('shop_location')
+        if machineid == machine.id:
+            # We can short-circuit arcade lookup because their machine
+            # is the current machine.
+            return True
+
+        their_machine = self.get_machine_by_id(machineid)
+        if their_machine is None:
+            return False
+
+        # The machine they joined matches the arcade of the current machine
+        return their_machine.arcade == machine.arcade
+
+    def get_ghost(
+        self,
+        ghost_type: int,
+        parameter: str,
+        ghost_length: int,
+        musicid: int,
+        chart: int,
+        userid: UserID,
+    ) -> Optional[Dict[str, Any]]:
+        ghost_score: Dict[str, Any] = None
+
+        if ghost_type == self.GHOST_TYPE_RIVAL:
+            rival_extid = int(parameter)
+            rival_userid = self.data.remote.user.from_extid(self.game, self.version, rival_extid)
+            if rival_userid is not None:
+                rival_profile = self.get_profile(rival_userid)
+                rival_score = self.data.remote.music.get_score(self.game, self.music_version, rival_userid, musicid, chart)
+                if rival_score is not None and rival_profile is not None:
+                    ghost_score = {
+                        'score': rival_score.points,
+                        'ghost': rival_score.data.get_bytes('ghost'),
+                        'name': rival_profile.get_str('name'),
+                        'pid': rival_profile.get_int('pid'),
+                    }
+
+        if (
+            ghost_type == self.GHOST_TYPE_GLOBAL_TOP or
+            ghost_type == self.GHOST_TYPE_LOCAL_TOP or
+            ghost_type == self.GHOST_TYPE_GLOBAL_AVERAGE or
+            ghost_type == self.GHOST_TYPE_LOCAL_AVERAGE
+        ):
+            if (
+                ghost_type == self.GHOST_TYPE_LOCAL_TOP or
+                ghost_type == self.GHOST_TYPE_LOCAL_AVERAGE
+            ):
+                all_scores = sorted(
+                    self.data.local.music.get_all_scores(game=self.game, version=self.music_version, songid=musicid, songchart=chart),
+                    key=lambda s: s[1].points,
+                    reverse=True,
+                )
+
+                # Figure out what arcade this user joined and filter scores by
+                # other users who have also joined that arcade.
+                my_profile = self.get_profile(userid)
+                if my_profile is None:
+                    my_profile = ValidatedDict()
+
+                if 'shop_location' in my_profile:
+                    shop_id = my_profile.get_int('shop_location')
+                    machine = self.get_machine_by_id(shop_id)
+                else:
+                    machine = None
+
+                if machine is not None:
+                    all_scores = [
+                        score for score in all_scores
+                        if self.user_joined_arcade(machine, self.get_any_profile(score[0]))
+                    ]
+                else:
+                    # Not joined an arcade, so nobody matches our scores
+                    all_scores = []
+            else:
+                all_scores = sorted(
+                    self.data.remote.music.get_all_scores(game=self.game, version=self.music_version, songid=musicid, songchart=chart),
+                    key=lambda s: s[1].points,
+                    reverse=True,
+                )
+
+            if (
+                ghost_type == self.GHOST_TYPE_GLOBAL_TOP or
+                ghost_type == self.GHOST_TYPE_LOCAL_TOP
+            ):
+                for potential_top in all_scores:
+                    top_userid = potential_top[0]
+                    top_score = potential_top[1]
+                    top_profile = self.get_any_profile(top_userid)
+                    if top_profile is not None:
+                        ghost_score = {
+                            'score': top_score.points,
+                            'ghost': top_score.data.get_bytes('ghost'),
+                            'name': top_profile.get_str('name'),
+                            'pid': top_profile.get_int('pid'),
+                            'extid': top_profile.get_int('extid'),
+                        }
+                        break
+
+            if (
+                ghost_type == self.GHOST_TYPE_GLOBAL_AVERAGE or
+                ghost_type == self.GHOST_TYPE_LOCAL_AVERAGE
+            ):
+                average_score, delta_ghost = self.delta_score([score[1] for score in all_scores], ghost_length)
+                if average_score is not None and delta_ghost is not None:
+                    ghost_score = {
+                        'score': average_score,
+                        'ghost': bytes([0] * ghost_length),
+                    }
+
+        if (
+            ghost_type == self.GHOST_TYPE_DAN_TOP or
+            ghost_type == self.GHOST_TYPE_DAN_AVERAGE
+        ):
+            is_dp = chart not in [
+                self.CHART_TYPE_N7,
+                self.CHART_TYPE_H7,
+                self.CHART_TYPE_A7,
+            ]
+            my_profile = self.get_profile(userid)
+            if my_profile is None:
+                my_profile = ValidatedDict()
+            if is_dp:
+                dan_rank = my_profile.get_int(self.DAN_RANKING_DOUBLE, -1)
+            else:
+                dan_rank = my_profile.get_int(self.DAN_RANKING_SINGLE, -1)
+
+            if dan_rank != -1:
+                all_scores = sorted(
+                    self.data.local.music.get_all_scores(game=self.game, version=self.music_version, songid=musicid, songchart=chart),
+                    key=lambda s: s[1].points,
+                    reverse=True,
+                )
+                all_profiles = self.data.local.user.get_all_profiles(self.game, self.version)
+                relevant_userids = {
+                    profile[0] for profile in all_profiles
+                    if profile[1].get_int(self.DAN_RANKING_DOUBLE if is_dp else self.DAN_RANKING_SINGLE) == dan_rank
+                }
+                relevant_scores = [
+                    score for score in all_scores
+                    if score[0] in relevant_userids
+                ]
+                if ghost_type == self.GHOST_TYPE_DAN_TOP:
+                    for potential_top in relevant_scores:
+                        top_userid = potential_top[0]
+                        top_score = potential_top[1]
+                        top_profile = self.get_any_profile(top_userid)
+                        if top_profile is not None:
+                            ghost_score = {
+                                'score': top_score.points,
+                                'ghost': top_score.data.get_bytes('ghost'),
+                                'name': top_profile.get_str('name'),
+                                'pid': top_profile.get_int('pid'),
+                                'extid': top_profile.get_int('extid'),
+                            }
+                            break
+
+                if ghost_type == self.GHOST_TYPE_DAN_AVERAGE:
+                    average_score, delta_ghost = self.delta_score([score[1] for score in relevant_scores], ghost_length)
+                    if average_score is not None and delta_ghost is not None:
+                        ghost_score = {
+                            'score': average_score,
+                            'ghost': bytes([0] * ghost_length),
+                        }
+
+        if (
+            ghost_type == self.GHOST_TYPE_RIVAL_TOP or
+            ghost_type == self.GHOST_TYPE_RIVAL_AVERAGE
+        ):
+            rival_extids = [int(e[1:-1]) for e in parameter.split(',')]
+            rival_userids = [
+                self.data.remote.user.from_extid(self.game, self.version, rival_extid)
+                for rival_extid in rival_extids
+            ]
+
+            all_scores = sorted(
+                [
+                    score for score in
+                    self.data.remote.music.get_all_scores(game=self.game, version=self.music_version, songid=musicid, songchart=chart)
+                    if score[0] in rival_userids
+                ],
+                key=lambda s: s[1].points,
+                reverse=True,
+            )
+            if ghost_type == self.GHOST_TYPE_RIVAL_TOP:
+                for potential_top in all_scores:
+                    top_userid = potential_top[0]
+                    top_score = potential_top[1]
+                    top_profile = self.get_any_profile(top_userid)
+                    if top_profile is not None:
+                        ghost_score = {
+                            'score': top_score.points,
+                            'ghost': top_score.data.get_bytes('ghost'),
+                            'name': top_profile.get_str('name'),
+                            'pid': top_profile.get_int('pid'),
+                            'extid': top_profile.get_int('extid'),
+                        }
+                        break
+
+            if ghost_type == self.GHOST_TYPE_RIVAL_AVERAGE:
+                average_score, delta_ghost = self.delta_score([score[1] for score in all_scores], ghost_length)
+                if average_score is not None and delta_ghost is not None:
+                    ghost_score = {
+                        'score': average_score,
+                        'ghost': bytes([0] * ghost_length),
+                    }
+
+        return ghost_score

bemani/backend/iidx/cannonballers.py

@@ -0,0 +1,15 @@
+# vim: set fileencoding=utf-8
+from typing import Optional
+
+from bemani.backend.iidx.base import IIDXBase
+from bemani.backend.iidx.sinobuz import IIDXSinobuz
+from bemani.common import VersionConstants
+
+
+class IIDXCannonBallers(IIDXBase):
+
+    name = 'Beatmania IIDX CANNON BALLERS'
+    version = VersionConstants.IIDX_CANNON_BALLERS
+
+    def previous_version(self) -> Optional[IIDXBase]:
+        return IIDXSinobuz(self.data, self.config, self.model)

bemani/backend/iidx/course.py

@@ -0,0 +1,72 @@
+# vim: set fileencoding=utf-8
+from typing import Tuple
+
+from bemani.backend.iidx.base import IIDXBase
+from bemani.common import ValidatedDict
+from bemani.data import UserID
+
+
+class IIDXCourse(IIDXBase):
+
+    COURSE_TYPE_SECRET = 'secret_course'
+    COURSE_TYPE_INTERNET_RANKING = 'ir_course'
+    COURSE_TYPE_CLASSIC = 'classic_course'
+
+    def id_and_chart_from_courseid(self, courseid: int) -> Tuple[int, int]:
+        return (int(courseid / 6), courseid % 6)
+
+    def update_course(
+        self,
+        userid: UserID,
+        coursetype: str,
+        courseid: int,
+        chart: int,
+        clear_status: int,
+        pgreats: int,
+        greats: int,
+    ) -> None:
+        # Range check course type
+        if coursetype not in [
+            self.COURSE_TYPE_SECRET,
+            self.COURSE_TYPE_INTERNET_RANKING,
+            self.COURSE_TYPE_CLASSIC,
+        ]:
+            raise Exception("Invalid course type value {}".format(coursetype))
+
+        # Range check medals
+        if clear_status not in [
+            self.CLEAR_STATUS_NO_PLAY,
+            self.CLEAR_STATUS_FAILED,
+            self.CLEAR_STATUS_ASSIST_CLEAR,
+            self.CLEAR_STATUS_EASY_CLEAR,
+            self.CLEAR_STATUS_CLEAR,
+            self.CLEAR_STATUS_HARD_CLEAR,
+            self.CLEAR_STATUS_EX_HARD_CLEAR,
+            self.CLEAR_STATUS_FULL_COMBO,
+        ]:
+            raise Exception("Invalid clear status value {}".format(clear_status))
+
+        # Update achievement to track course statistics
+        course_score = self.data.local.user.get_achievement(
+            self.game,
+            self.version,
+            userid,
+            courseid * 6 + chart,
+            coursetype,
+        )
+        if course_score is None:
+            course_score = ValidatedDict()
+        course_score.replace_int('clear_status', max(clear_status, course_score.get_int('clear_status')))
+        old_ex_score = (course_score.get_int('pgnum') * 2) + course_score.get_int('gnum')
+        if old_ex_score < ((pgreats * 2) + greats):
+            course_score.replace_int('pgnum', pgreats)
+            course_score.replace_int('gnum', greats)
+
+        self.data.local.user.put_achievement(
+            self.game,
+            self.version,
+            userid,
+            courseid * 6 + chart,
+            coursetype,
+            course_score,
+        )

bemani/backend/iidx/factory.py

@@ -0,0 +1,133 @@
+from typing import Dict, Optional, Any
+
+from bemani.backend.base import Base, Factory
+from bemani.backend.iidx.stubs import (
+    IIDX1stStyle,
+    IIDX2ndStyle,
+    IIDX3rdStyle,
+    IIDX4thStyle,
+    IIDX5thStyle,
+    IIDX6thStyle,
+    IIDX7thStyle,
+    IIDX8thStyle,
+    IIDX9thStyle,
+    IIDX10thStyle,
+    IIDXRed,
+    IIDXHappySky,
+    IIDXDistorted,
+    IIDXGold,
+    IIDXDJTroopers,
+    IIDXEmpress,
+    IIDXSirius,
+    IIDXResortAnthem,
+    IIDXLincle,
+)
+from bemani.backend.iidx.tricoro import IIDXTricoro
+from bemani.backend.iidx.spada import IIDXSpada
+from bemani.backend.iidx.pendual import IIDXPendual
+from bemani.backend.iidx.copula import IIDXCopula
+from bemani.backend.iidx.sinobuz import IIDXSinobuz
+from bemani.backend.iidx.cannonballers import IIDXCannonBallers
+from bemani.common import Model, VersionConstants
+from bemani.data import Data
+
+
+class IIDXFactory(Factory):
+
+    MANAGED_CLASSES = [
+        IIDX1stStyle,
+        IIDX2ndStyle,
+        IIDX3rdStyle,
+        IIDX4thStyle,
+        IIDX5thStyle,
+        IIDX6thStyle,
+        IIDX7thStyle,
+        IIDX8thStyle,
+        IIDX9thStyle,
+        IIDX10thStyle,
+        IIDXRed,
+        IIDXHappySky,
+        IIDXDistorted,
+        IIDXGold,
+        IIDXDJTroopers,
+        IIDXEmpress,
+        IIDXSirius,
+        IIDXResortAnthem,
+        IIDXLincle,
+        IIDXTricoro,
+        IIDXSpada,
+        IIDXPendual,
+        IIDXCopula,
+        IIDXSinobuz,
+        IIDXCannonBallers,
+    ]
+
+    @classmethod
+    def register_all(cls) -> None:
+        for game in ['JDJ', 'JDZ', 'KDZ', 'LDJ']:
+            Base.register(game, IIDXFactory)
+
+    @classmethod
+    def create(cls, data: Data, config: Dict[str, Any], model: Model, parentmodel: Optional[Model]=None) -> Optional[Base]:
+
+        def version_from_date(date: int) -> Optional[int]:
+            if date < 2013100200:
+                return VersionConstants.IIDX_TRICORO
+            if date >= 2013100200 and date < 2014091700:
+                return VersionConstants.IIDX_SPADA
+            if date >= 2014091700 and date < 2015111100:
+                return VersionConstants.IIDX_PENDUAL
+            if date >= 2015111100 and date < 2016102600:
+                return VersionConstants.IIDX_COPULA
+            if date >= 2016102600 and date < 2017122100:
+                return VersionConstants.IIDX_SINOBUZ
+            if date >= 2017122100:
+                return VersionConstants.IIDX_CANNON_BALLERS
+            return None
+
+        if model.game == 'JDJ':
+            return IIDXSirius(data, config, model)
+        if model.game == 'JDZ':
+            return IIDXResortAnthem(data, config, model)
+        if model.game == 'KDZ':
+            return IIDXLincle(data, config, model)
+        if model.game == 'LDJ':
+            if model.version is None:
+                if parentmodel is None:
+                    return None
+
+                # We have no way to tell apart newer versions. However, we can make
+                # an educated guess if we happen to be summoned for old profile lookup.
+                if parentmodel.game not in ['JDJ', 'JDZ', 'KDZ', 'LDJ']:
+                    return None
+                parentversion = version_from_date(parentmodel.version)
+                if parentversion == VersionConstants.IIDX_SPADA:
+                    return IIDXTricoro(data, config, model)
+                if parentversion == VersionConstants.IIDX_PENDUAL:
+                    return IIDXSpada(data, config, model)
+                if parentversion == VersionConstants.IIDX_COPULA:
+                    return IIDXPendual(data, config, model)
+                if parentversion == VersionConstants.IIDX_SINOBUZ:
+                    return IIDXCopula(data, config, model)
+                if parentversion == VersionConstants.IIDX_CANNON_BALLERS:
+                    return IIDXSinobuz(data, config, model)
+
+                # Unknown older version
+                return None
+
+            version = version_from_date(model.version)
+            if version == VersionConstants.IIDX_TRICORO:
+                return IIDXTricoro(data, config, model)
+            if version == VersionConstants.IIDX_SPADA:
+                return IIDXSpada(data, config, model)
+            if version == VersionConstants.IIDX_PENDUAL:
+                return IIDXPendual(data, config, model)
+            if version == VersionConstants.IIDX_COPULA:
+                return IIDXCopula(data, config, model)
+            if version == VersionConstants.IIDX_SINOBUZ:
+                return IIDXSinobuz(data, config, model)
+            if version == VersionConstants.IIDX_CANNON_BALLERS:
+                return IIDXCannonBallers(data, config, model)
+
+        # Unknown game version
+        return None

bemani/backend/iidx/stubs.py

@@ -0,0 +1,173 @@
+# vim: set fileencoding=utf-8
+from typing import Optional
+
+from bemani.backend.iidx.base import IIDXBase
+from bemani.common import VersionConstants
+
+
+class IIDX1stStyle(IIDXBase):
+
+    name = 'Beatmania IIDX 1st style & substream'
+    version = VersionConstants.IIDX
+
+
+class IIDX2ndStyle(IIDXBase):
+
+    name = 'Beatmania IIDX 2nd style'
+    version = VersionConstants.IIDX_2ND_STYLE
+
+    def previous_version(self) -> Optional[IIDXBase]:
+        return IIDX1stStyle(self.data, self.config, self.model)
+
+
+class IIDX3rdStyle(IIDXBase):
+
+    name = 'Beatmania IIDX 3rd style'
+    version = VersionConstants.IIDX_3RD_STYLE
+
+    def previous_version(self) -> Optional[IIDXBase]:
+        return IIDX2ndStyle(self.data, self.config, self.model)
+
+
+class IIDX4thStyle(IIDXBase):
+
+    name = 'Beatmania IIDX 4th style'
+    version = VersionConstants.IIDX_4TH_STYLE
+
+    def previous_version(self) -> Optional[IIDXBase]:
+        return IIDX3rdStyle(self.data, self.config, self.model)
+
+
+class IIDX5thStyle(IIDXBase):
+
+    name = 'Beatmania IIDX 5th style'
+    version = VersionConstants.IIDX_5TH_STYLE
+
+    def previous_version(self) -> Optional[IIDXBase]:
+        return IIDX4thStyle(self.data, self.config, self.model)
+
+
+class IIDX6thStyle(IIDXBase):
+
+    name = 'Beatmania IIDX 6th style'
+    version = VersionConstants.IIDX_6TH_STYLE
+
+    def previous_version(self) -> Optional[IIDXBase]:
+        return IIDX5thStyle(self.data, self.config, self.model)
+
+
+class IIDX7thStyle(IIDXBase):
+
+    name = 'Beatmania IIDX 7th style'
+    version = VersionConstants.IIDX_7TH_STYLE
+
+    def previous_version(self) -> Optional[IIDXBase]:
+        return IIDX6thStyle(self.data, self.config, self.model)
+
+
+class IIDX8thStyle(IIDXBase):
+
+    name = 'Beatmania IIDX 8th style'
+    version = VersionConstants.IIDX_8TH_STYLE
+
+    def previous_version(self) -> Optional[IIDXBase]:
+        return IIDX7thStyle(self.data, self.config, self.model)
+
+
+class IIDX9thStyle(IIDXBase):
+
+    name = 'Beatmania IIDX 9th style'
+    version = VersionConstants.IIDX_9TH_STYLE
+
+    def previous_version(self) -> Optional[IIDXBase]:
+        return IIDX8thStyle(self.data, self.config, self.model)
+
+
+class IIDX10thStyle(IIDXBase):
+
+    name = 'Beatmania IIDX 10th style'
+    version = VersionConstants.IIDX_10TH_STYLE
+
+    def previous_version(self) -> Optional[IIDXBase]:
+        return IIDX9thStyle(self.data, self.config, self.model)
+
+
+class IIDXRed(IIDXBase):
+
+    name = 'Beatmania IIDX RED'
+    version = VersionConstants.IIDX_RED
+
+    def previous_version(self) -> Optional[IIDXBase]:
+        return IIDX10thStyle(self.data, self.config, self.model)
+
+
+class IIDXHappySky(IIDXBase):
+
+    name = 'Beatmania IIDX HAPPY SKY'
+    version = VersionConstants.IIDX_HAPPY_SKY
+
+    def previous_version(self) -> Optional[IIDXBase]:
+        return IIDXRed(self.data, self.config, self.model)
+
+
+class IIDXDistorted(IIDXBase):
+
+    name = 'Beatmania IIDX DistorteD'
+    version = VersionConstants.IIDX_DISTORTED
+
+    def previous_version(self) -> Optional[IIDXBase]:
+        return IIDXHappySky(self.data, self.config, self.model)
+
+
+class IIDXGold(IIDXBase):
+
+    name = 'Beatmania IIDX GOLD'
+    version = VersionConstants.IIDX_GOLD
+
+    def previous_version(self) -> Optional[IIDXBase]:
+        return IIDXDistorted(self.data, self.config, self.model)
+
+
+class IIDXDJTroopers(IIDXBase):
+
+    name = 'Beatmania IIDX DJ TROOPERS'
+    version = VersionConstants.IIDX_DJ_TROOPERS
+
+    def previous_version(self) -> Optional[IIDXBase]:
+        return IIDXGold(self.data, self.config, self.model)
+
+
+class IIDXEmpress(IIDXBase):
+
+    name = 'Beatmania IIDX EMPRESS'
+    version = VersionConstants.IIDX_EMPRESS
+
+    def previous_version(self) -> Optional[IIDXBase]:
+        return IIDXDJTroopers(self.data, self.config, self.model)
+
+
+class IIDXSirius(IIDXBase):
+
+    name = 'Beatmania IIDX SIRIUS'
+    version = VersionConstants.IIDX_SIRIUS
+
+    def previous_version(self) -> Optional[IIDXBase]:
+        return IIDXEmpress(self.data, self.config, self.model)
+
+
+class IIDXResortAnthem(IIDXBase):
+
+    name = 'Beatmania IIDX Resort Anthem'
+    version = VersionConstants.IIDX_RESORT_ANTHEM
+
+    def previous_version(self) -> Optional[IIDXBase]:
+        return IIDXSirius(self.data, self.config, self.model)
+
+
+class IIDXLincle(IIDXBase):
+
+    name = 'Beatmania IIDX Lincle'
+    version = VersionConstants.IIDX_LINCLE
+
+    def previous_version(self) -> Optional[IIDXBase]:
+        return IIDXResortAnthem(self.data, self.config, self.model)

bemani/backend/jubeat/__init__.py

@@ -0,0 +1,2 @@
+from bemani.backend.jubeat.factory import JubeatFactory
+from bemani.backend.jubeat.base import JubeatBase

bemani/backend/jubeat/base.py

@@ -0,0 +1,255 @@
+# vim: set fileencoding=utf-8
+from typing import Dict, List, Optional
+
+from bemani.backend.base import Base
+from bemani.backend.core import CoreHandler, CardManagerHandler, PASELIHandler
+from bemani.common import DBConstants, GameConstants, ValidatedDict
+from bemani.data import Score, UserID
+from bemani.protocol import Node
+
+
+class JubeatBase(CoreHandler, CardManagerHandler, PASELIHandler, Base):
+    """
+    Base game class for all Jubeat versions. Handles common functionality for getting
+    profiles based on refid, creating new profiles, looking up and saving scores.
+    """
+
+    game = GameConstants.JUBEAT
+
+    GAME_FLAG_BIT_PLAYED = 0x1
+    GAME_FLAG_BIT_CLEARED = 0x2
+    GAME_FLAG_BIT_FULL_COMBO = 0x4
+    GAME_FLAG_BIT_EXCELLENT = 0x8
+    GAME_FLAG_BIT_NEARLY_FULL_COMBO = 0x10
+    GAME_FLAG_BIT_NEARLY_EXCELLENT = 0x20
+    GAME_FLAG_BIT_NO_GRAY = 0x40
+    GAME_FLAG_BIT_NO_YELLOW = 0x80
+
+    PLAY_MEDAL_FAILED = DBConstants.JUBEAT_PLAY_MEDAL_FAILED
+    PLAY_MEDAL_CLEARED = DBConstants.JUBEAT_PLAY_MEDAL_CLEARED
+    PLAY_MEDAL_NEARLY_FULL_COMBO = DBConstants.JUBEAT_PLAY_MEDAL_NEARLY_FULL_COMBO
+    PLAY_MEDAL_FULL_COMBO = DBConstants.JUBEAT_PLAY_MEDAL_FULL_COMBO
+    PLAY_MEDAL_NEARLY_EXCELLENT = DBConstants.JUBEAT_PLAY_MEDAL_NEARLY_EXCELLENT
+    PLAY_MEDAL_EXCELLENT = DBConstants.JUBEAT_PLAY_MEDAL_EXCELLENT
+
+    CHART_TYPE_BASIC = 0
+    CHART_TYPE_ADVANCED = 1
+    CHART_TYPE_EXTREME = 2
+
+    def previous_version(self) -> Optional['JubeatBase']:
+        """
+        Returns the previous version of the game, based on this game. Should
+        be overridden.
+        """
+        return None
+
+    def put_profile(self, userid: UserID, profile: ValidatedDict) -> None:
+        """
+        Save a new profile for this user given a game/version. Overrides but calls
+        the same functionality in Base, to ensure we don't save calculated values.
+
+        Parameters:
+            userid - The user ID we are saving the profile for.
+            profile - A dictionary that should be looked up later using get_profile.
+        """
+        if 'has_old_version' in profile:
+            del profile['has_old_version']
+        super().put_profile(userid, profile)
+
+    def format_profile(self, userid: UserID, profile: ValidatedDict) -> Node:
+        """
+        Base handler for a profile. Given a userid and a profile dictionary,
+        return a Node representing a profile. Should be overridden.
+        """
+        return Node.void('gametop')
+
+    def format_scores(self, userid: UserID, profile: ValidatedDict, scores: List[Score]) -> Node:
+        """
+        Base handler for a score list. Given a userid, profile and a score list,
+        return a Node representing a score list. Should be overridden.
+        """
+        return Node.void('gametop')
+
+    def unformat_profile(self, userid: UserID, request: Node, oldprofile: ValidatedDict) -> ValidatedDict:
+        """
+        Base handler for profile parsing. Given a request and an old profile,
+        return a new profile that's been updated with the contents of the request.
+        Should be overridden.
+        """
+        return oldprofile
+
+    def get_profile_by_refid(self, refid: Optional[str]) -> Optional[Node]:
+        """
+        Given a RefID, return a formatted profile node. Basically every game
+        needs a profile lookup, even if it handles where that happens in
+        a different request. This is provided for code deduplication.
+        """
+        if refid is None:
+            return None
+
+        # First try to load the actual profile
+        userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        profile = self.get_profile(userid)
+        if profile is None:
+            return None
+
+        # Now try to find out if the profile is new or old
+        oldversion = self.previous_version()
+        oldprofile = oldversion.get_profile(userid)
+        profile['has_old_version'] = oldprofile is not None
+
+        # Now, return it
+        return self.format_profile(userid, profile)
+
+    def new_profile_by_refid(self, refid: Optional[str], name: Optional[str]) -> Node:
+        """
+        Given a RefID and an optional name, create a profile and then return
+        a formatted profile node. Similar rationale to get_profile_by_refid.
+        """
+        if refid is None:
+            return None
+
+        if name is None:
+            name = 'なし'
+
+        # First, create and save the default profile
+        userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        defaultprofile = ValidatedDict({
+            'name': name,
+        })
+        self.put_profile(userid, defaultprofile)
+
+        # Now, reload and format the profile, looking up the has old version flag
+        profile = self.get_profile(userid)
+
+        oldversion = self.previous_version()
+        oldprofile = oldversion.get_profile(userid)
+        profile['has_old_version'] = oldprofile is not None
+
+        return self.format_profile(userid, profile)
+
+    def get_scores_by_extid(self, extid: Optional[int]) -> Optional[Node]:
+        """
+        Given an ExtID, return a formatted score node. Similar rationale to
+        get_profile_by_refid.
+        """
+        if extid is None:
+            return None
+
+        userid = self.data.remote.user.from_extid(self.game, self.version, extid)
+        scores = self.data.remote.music.get_scores(self.game, self.version, userid)
+        if scores is None:
+            return None
+        profile = self.get_profile(userid)
+        if profile is None:
+            return None
+        return self.format_scores(userid, profile, scores)
+
+    def update_score(
+        self,
+        userid: UserID,
+        timestamp: int,
+        songid: int,
+        chart: int,
+        points: int,
+        medal: int,
+        combo: int,
+        ghost: Optional[List[int]]=None,
+        stats: Optional[Dict[str, int]]=None,
+    ) -> None:
+        """
+        Given various pieces of a score, update the user's high score and score
+        history in a controlled manner, so all games in Jubeat series can expect
+        the same attributes in a score.
+        """
+        # Range check medals
+        if medal not in [
+            self.PLAY_MEDAL_FAILED,
+            self.PLAY_MEDAL_CLEARED,
+            self.PLAY_MEDAL_NEARLY_FULL_COMBO,
+            self.PLAY_MEDAL_FULL_COMBO,
+            self.PLAY_MEDAL_NEARLY_EXCELLENT,
+            self.PLAY_MEDAL_EXCELLENT,
+        ]:
+            raise Exception("Invalid medal value {}".format(medal))
+
+        oldscore = self.data.local.music.get_score(
+            self.game,
+            self.version,
+            userid,
+            songid,
+            chart,
+        )
+
+        # Score history is verbatum, instead of highest score
+        history = ValidatedDict({})
+        oldpoints = points
+
+        if oldscore is None:
+            # If it is a new score, create a new dictionary to add to
+            scoredata = ValidatedDict({})
+            raised = True
+            highscore = True
+        else:
+            # Set the score to any new record achieved
+            raised = points > oldscore.points
+            highscore = points >= oldscore.points
+            points = max(oldscore.points, points)
+            scoredata = oldscore.data
+
+        # Replace medal with highest value
+        scoredata.replace_int('medal', max(scoredata.get_int('medal'), medal))
+        history.replace_int('medal', medal)
+
+        # Increment counters based on medal
+        if medal == self.PLAY_MEDAL_CLEARED:
+            scoredata.increment_int('clear_count')
+        if medal == self.PLAY_MEDAL_FULL_COMBO:
+            scoredata.increment_int('full_combo_count')
+        if medal == self.PLAY_MEDAL_EXCELLENT:
+            scoredata.increment_int('excellent_count')
+
+        # If we have a combo, replace it
+        scoredata.replace_int('combo', max(scoredata.get_int('combo'), combo))
+        history.replace_int('combo', combo)
+
+        if stats is not None:
+            if raised:
+                # We have stats, and there's a new high score, update the stats
+                scoredata.replace_dict('stats', stats)
+            history.replace_dict('stats', stats)
+
+        if ghost is not None:
+            # Update the ghost regardless, but don't bother with it in history
+            scoredata.replace_int_array('ghost', len(ghost), ghost)
+
+        # Look up where this score was earned
+        lid = self.get_machine_id()
+
+        # Write the new score back
+        self.data.local.music.put_score(
+            self.game,
+            self.version,
+            userid,
+            songid,
+            chart,
+            lid,
+            points,
+            scoredata,
+            highscore,
+            timestamp=timestamp,
+        )
+
+        # Save the history of this score too
+        self.data.local.music.put_attempt(
+            self.game,
+            self.version,
+            userid,
+            songid,
+            chart,
+            lid,
+            oldpoints,
+            history,
+            raised,
+            timestamp=timestamp,
+        )

bemani/backend/jubeat/common.py

@@ -0,0 +1,130 @@
+import time
+
+from bemani.backend.jubeat.base import JubeatBase
+from bemani.protocol import Node
+
+
+class JubeatLoggerReportHandler(JubeatBase):
+
+    def handle_logger_report_request(self, request: Node) -> Node:
+        # Handle this by returning nothing, game doesn't care
+        root = Node.void('logger')
+        return root
+
+
+class JubeatDemodataGetNewsHandler(JubeatBase):
+
+    def handle_demodata_get_news_request(self, request: Node) -> Node:
+        demodata = Node.void('demodata')
+        data = Node.void('data')
+        demodata.add_child(data)
+
+        officialnews = Node.void('officialnews')
+        data.add_child(officialnews)
+        officialnews.set_attribute('count', '0')
+
+        return demodata
+
+
+class JubeatDemodataGetHitchartHandler(JubeatBase):
+
+    def handle_demodata_get_hitchart_request(self, request: Node) -> Node:
+        demodata = Node.void('demodata')
+        data = Node.void('data')
+        demodata.add_child(data)
+
+        # Not sure what this is, maybe date?
+        data.add_child(Node.string('update', time.strftime("%d/%m/%Y")))
+
+        # No idea which songs are licensed or regular, so only return hit chart
+        # for all songs on regular mode.
+        hitchart_lic = Node.void('hitchart_lic')
+        data.add_child(hitchart_lic)
+        hitchart_lic.set_attribute('count', '0')
+
+        songs = self.data.local.music.get_hit_chart(self.game, self.version, 10)
+        hitchart_org = Node.void('hitchart_org')
+        data.add_child(hitchart_org)
+        hitchart_org.set_attribute('count', str(len(songs)))
+        rank = 1
+        for song in songs:
+            rankdata = Node.void('rankdata')
+            hitchart_org.add_child(rankdata)
+            rankdata.add_child(Node.s32('music_id', song[0]))
+            rankdata.add_child(Node.s16('rank', rank))
+            rankdata.add_child(Node.s16('prev', rank))
+            rank = rank + 1
+
+        return demodata
+
+
+class JubeatLobbyCheckHandler(JubeatBase):
+
+    def handle_lobby_check_request(self, request: Node) -> Node:
+        root = Node.void('lobby')
+        data = Node.void('data')
+        root.add_child(data)
+
+        data.add_child(Node.s16('interval', 0))
+        data.add_child(Node.s16('entry_timeout', 0))
+        entrant_nr = Node.u32('entrant_nr', 0)
+        entrant_nr.set_attribute('time', '0')
+        data.add_child(entrant_nr)
+
+        return root
+
+
+class JubeatGamendRegisterHandler(JubeatBase):
+
+    def handle_gameend_regist_request(self, request: Node) -> Node:
+        data = request.child('data')
+        player = data.child('player')
+
+        if player is not None:
+            refid = player.child_value('refid')
+        else:
+            refid = None
+
+        if refid is not None:
+            userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        else:
+            userid = None
+
+        if userid is not None:
+            oldprofile = self.get_profile(userid)
+            newprofile = self.unformat_profile(userid, request, oldprofile)
+        else:
+            newprofile = None
+
+        if userid is not None and newprofile is not None:
+            self.put_profile(userid, newprofile)
+
+        gameend = Node.void('gameend')
+        data = Node.void('data')
+        gameend.add_child(data)
+        player = Node.void('player')
+        data.add_child(player)
+        player.add_child(Node.s32('session_id', 1))
+        player.add_child(Node.s32('end_final_session_id', 1))
+        return gameend
+
+
+class JubeatGametopGetMeetingHandler(JubeatBase):
+
+    def handle_gametop_get_meeting_request(self, request: Node) -> Node:
+        gametop = Node.void('gametop')
+        data = Node.void('data')
+        gametop.add_child(data)
+        meeting = Node.void('meeting')
+        data.add_child(meeting)
+        single = Node.void('single')
+        meeting.add_child(single)
+        single.set_attribute('count', '0')
+        tag = Node.void('tag')
+        meeting.add_child(tag)
+        tag.set_attribute('count', '0')
+        reward = Node.void('reward')
+        data.add_child(reward)
+        reward.add_child(Node.s32('total', -1))
+        reward.add_child(Node.s32('point', -1))
+        return gametop

bemani/backend/jubeat/course.py

@@ -0,0 +1,541 @@
+# vim: set fileencoding=utf-8
+from typing import Any, Dict, List
+
+from bemani.data import UserID
+from bemani.backend.jubeat.base import JubeatBase
+
+
+class JubeatCourse(JubeatBase):
+
+    COURSE_RATING_FAILED = 100
+    COURSE_RATING_BRONZE = 200
+    COURSE_RATING_SILVER = 300
+    COURSE_RATING_GOLD = 400
+
+    COURSE_REQUIREMENT_SCORE = 100
+    COURSE_REQUIREMENT_FULL_COMBO = 200
+    COURSE_REQUIREMENT_PERFECT_PERCENT = 300
+
+    def get_all_courses(self) -> List[Dict[str, Any]]:
+        # List of base courses for Saucer Fulfill+ from BemaniWiki
+        return [
+            {
+                'id': 1,
+                'name': '溢れ出した記憶、特別なあなたにありがとう。',
+                'level': 1,
+                'music': [
+                    (50000241, 2),
+                    (10000052, 2),
+                    (30000042, 2),
+                    (50000085, 2),
+                    (50000144, 2),
+                ],
+                'requirements': {
+                    self.COURSE_REQUIREMENT_SCORE: [850000, 900000, 950000],
+                    self.COURSE_REQUIREMENT_FULL_COMBO: [0, 1, 2],
+                },
+            },
+            {
+                'id': 2,
+                'name': 'コースモードが怖い？ばっかお前TAGがついてるだろ',
+                'level': 1,
+                'music': [
+                    (50000121, 1),
+                    (30000122, 1),
+                    (40000159, 1),
+                    (50000089, 1),
+                    (40000051, 2),
+                ],
+                'requirements': {
+                    self.COURSE_REQUIREMENT_SCORE: [800000, 850000, 900000],
+                    self.COURSE_REQUIREMENT_FULL_COMBO: [0, 1, 2],
+                },
+            },
+            {
+                'id': 3,
+                'name': '満月の鐘踊り響くは虚空から成る恋の歌',
+                'level': 2,
+                'music': [
+                    (40000121, 2),
+                    (50000188, 2),
+                    (30000047, 2),
+                    (50000237, 2),
+                    (50000176, 2),
+                ],
+                'requirements': {
+                    self.COURSE_REQUIREMENT_SCORE: [850000, 900000, 950000],
+                    self.COURSE_REQUIREMENT_FULL_COMBO: [1, 2, 3],
+                },
+            },
+            {
+                'id': 4,
+                'name': 'スミスゼミナール 夏の陣開講記念 基本編',
+                'level': 2,
+                'music': [
+                    (50000267, 1),
+                    (50000233, 1),
+                    (50000228, 1),
+                    (50000268, 1),
+                    (50000291, 1),
+                ],
+                'requirements': {
+                    self.COURSE_REQUIREMENT_FULL_COMBO: [1, 2, 3],
+                    self.COURSE_REQUIREMENT_PERFECT_PERCENT: [85, 90, 95],
+                },
+            },
+            {
+                'id': 5,
+                'name': 'HARDモードじゃないから、絶対、大丈夫だよっ！',
+                'level': 2,
+                'music': [
+                    (50000144, 2),
+                    (50000188, 2),
+                    (50000070, 2),
+                    (50000151, 2),
+                    (50000152, 2),
+                ],
+                'requirements': {
+                    self.COURSE_REQUIREMENT_SCORE: [850000, 900000, 950000],
+                    self.COURSE_REQUIREMENT_FULL_COMBO: [0, 1, 2],
+                },
+            },
+            {
+                'id': 6,
+                'name': '星明かりの下、愛という名の日替わりランチを君と',
+                'level': 3,
+                'music': [
+                    (50000196, 1),
+                    (50000151, 2),
+                    (50000060, 1),
+                    (40000048, 2),
+                    (10000051, 2),
+                ],
+                'requirements': {
+                    self.COURSE_REQUIREMENT_PERFECT_PERCENT: [70, 80, 90],
+                },
+            },
+            {
+                'id': 7,
+                'name': '輝く北極星と幸せなヒーロー',
+                'level': 4,
+                'music': [
+                    (50000079, 2),
+                    (20000044, 2),
+                    (50000109, 2),
+                    (10000043, 2),
+                    (10000042, 2),
+                ],
+                'requirements': {
+                    self.COURSE_REQUIREMENT_SCORE: [900000, 950000, 980000],
+                    self.COURSE_REQUIREMENT_FULL_COMBO: [1, 2, 3],
+                },
+            },
+            {
+                'id': 8,
+                'name': '花-鳥-藻-夏',
+                'level': 4,
+                'music': [
+                    (10000068, 2),
+                    (40000154, 2),
+                    (50000123, 1),
+                    (40000051, 2),
+                    (30000045, 2),
+                ],
+                'requirements': {
+                    self.COURSE_REQUIREMENT_PERFECT_PERCENT: [70, 80, 90],
+                },
+            },
+            {
+                'id': 9,
+                'name': 'TAG生誕祭2014 俺の記録を抜いてみろ！',
+                'level': 4,
+                'music': [
+                    (30000122, 2),
+                    (50000086, 2),
+                    (50000121, 2),
+                    (50000196, 2),
+                    (40000051, 2),
+                ],
+                'requirements': {
+                    self.COURSE_REQUIREMENT_SCORE: [900000, 950000, 967252],
+                    self.COURSE_REQUIREMENT_FULL_COMBO: [0, 0, 1],
+                },
+            },
+            {
+                'id': 10,
+                'name': 'さよなら、亡くした恋と蝶の舞うヒストリア',
+                'level': 5,
+                'music': [
+                    (20000041, 2),
+                    (30000044, 2),
+                    (50000037, 2),
+                    (20000124, 2),
+                    (50000033, 2),
+                ],
+                'requirements': {
+                    self.COURSE_REQUIREMENT_PERFECT_PERCENT: [80, 85, 90],
+                },
+            },
+            {
+                'id': 11,
+                'name': 'きらきらほしふるまぼろしなぎさちゃん',
+                'level': 5,
+                'music': [
+                    (30000050, 2),
+                    (30000049, 2),
+                    (50000235, 2),
+                    (50000157, 2),
+                    (50000038, 2),
+                ],
+                'requirements': {
+                    self.COURSE_REQUIREMENT_SCORE: [700000, 800000, 900000],
+                },
+            },
+            {
+                'id': 12,
+                'name': 'The Memorial Third: 僕みたいに演奏してね',
+                'level': 5,
+                'music': [
+                    (10000037, 2),
+                    (20000048, 1),
+                    (50000253, 1),
+                    (20000121, 2),
+                    (50000133, 2),
+                ],
+                'requirements': {
+                    self.COURSE_REQUIREMENT_PERFECT_PERCENT: [75, 80, 85],
+                },
+            },
+            {
+                'id': 13,
+                'name': 'Enjoy! 4thKAC ~ Memories of saucer ~',
+                'level': 5,
+                'music': [
+                    (50000206, 1),
+                    (50000023, 1),
+                    (50000078, 1),
+                    (50000203, 1),
+                    (50000323, 1),
+                ],
+                'requirements': {
+                    self.COURSE_REQUIREMENT_SCORE: [900000, 950000, 980000],
+                    self.COURSE_REQUIREMENT_FULL_COMBO: [1, 2, 4],
+                },
+            },
+            {
+                'id': 14,
+                'name': '風に吹かれるキケンなシロクマダンス',
+                'level': 6,
+                'music': [
+                    (50000059, 2),
+                    (50000197, 2),
+                    (30000037, 2),
+                    (50000182, 2),
+                    (20000038, 2),
+                ],
+                'requirements': {
+                    self.COURSE_REQUIREMENT_SCORE: [900000, 950000, 980000],
+                    self.COURSE_REQUIREMENT_FULL_COMBO: [1, 2, 3],
+                },
+            },
+            {
+                'id': 15,
+                'name': '君主は視線で友との愛を語るめう',
+                'level': 6,
+                'music': [
+                    (40000052, 2),
+                    (50000152, 2),
+                    (50000090, 2),
+                    (20000040, 2),
+                    (50000184, 2),
+                ],
+                'requirements': {
+                    self.COURSE_REQUIREMENT_PERFECT_PERCENT: [85, 90, 95],
+                },
+            },
+            {
+                'id': 16,
+                'name': 'スミスゼミナール 夏の陣開講記念 応用編',
+                'level': 6,
+                'music': [
+                    (50000233, 2),
+                    (50000267, 2),
+                    (50000268, 2),
+                    (50000228, 2),
+                    (50000291, 2),
+                ],
+                'requirements': {
+                    self.COURSE_REQUIREMENT_SCORE: [750000, 850000, 900000],
+                },
+            },
+            {
+                'id': 17,
+                'name': '天から降り注ぐ星はまるで甘いキャンディ',
+                'level': 7,
+                'music': [
+                    (20000044, 2),
+                    (30000050, 2),
+                    (50000080, 2),
+                    (40000126, 2),
+                    (10000067, 2),
+                ],
+                'requirements': {
+                    self.COURSE_REQUIREMENT_PERFECT_PERCENT: [85, 90, 95],
+                },
+            },
+            {
+                'id': 18,
+                'name': 'てんとう虫が囁いている「Wow Wow…」',
+                'level': 7,
+                'music': [
+                    (50000132, 2),
+                    (40000128, 2),
+                    (10000036, 2),
+                    (50000119, 2),
+                    (50000030, 2),
+                ],
+                'requirements': {
+                    self.COURSE_REQUIREMENT_PERFECT_PERCENT: [85, 90, 95],
+                },
+            },
+            {
+                'id': 19,
+                'name': 'HARDモードでも大丈夫だよ！絶対、大丈夫だよっ！',
+                'level': 7,
+                'music': [
+                    (50000144, 2),
+                    (50000070, 2),
+                    (50000188, 2),
+                    (50000151, 2),
+                    (50000152, 2),
+                ],
+                'requirements': {
+                    self.COURSE_REQUIREMENT_SCORE: [850000, 900000, 950000],
+                },
+            },
+            {
+                'id': 20,
+                'name': 'こんなHARDモード、滅べばいい…',
+                'level': 7,
+                'music': [
+                    (50000294, 2),
+                    (50000295, 2),
+                    (50000234, 2),
+                    (50000245, 2),
+                    (50000282, 2),
+                ],
+                'requirements': {
+                    self.COURSE_REQUIREMENT_SCORE: [850000, 900000, 950000],
+                },
+            },
+            {
+                'id': 21,
+                'name': 'Challenge! 4thKAC ~ Memories of saucer ~',
+                'level': 7,
+                'music': [
+                    (50000206, 2),
+                    (50000023, 2),
+                    (50000078, 2),
+                    (50000203, 2),
+                    (50000323, 2),
+                ],
+                'requirements': {
+                    self.COURSE_REQUIREMENT_SCORE: [900000, 950000, 980000],
+                },
+            },
+            {
+                'id': 22,
+                'name': 'サヨナラ・キングコング ~ 恋のつぼみは愛の虹へ ~',
+                'level': 8,
+                'music': [
+                    (50000148, 2),
+                    (50000101, 2),
+                    (10000064, 2),
+                    (50000171, 2),
+                    (50000070, 2),
+                ],
+                'requirements': {
+                    self.COURSE_REQUIREMENT_SCORE: [900000, 950000, 980000],
+                },
+            },
+            {
+                'id': 23,
+                'name': '風に舞う白鳥の翼と花弁、さながら万華鏡のよう',
+                'level': 8,
+                'music': [
+                    (30000036, 2),
+                    (50000122, 2),
+                    (10000062, 2),
+                    (50000199, 2),
+                    (40000153, 2),
+                ],
+                'requirements': {
+                    self.COURSE_REQUIREMENT_PERFECT_PERCENT: [90, 95, 98],
+                },
+            },
+            {
+                'id': 24,
+                'name': 'The 小さなおぼろガチョウ♪',
+                'level': 8,
+                'music': [
+                    (50000049, 2),
+                    (50000071, 2),
+                    (10000041, 2),
+                    (50000031, 2),
+                    (40000129, 2),
+                ],
+                'requirements': {
+                    self.COURSE_REQUIREMENT_SCORE: [970000, 980000, 990000],
+                    self.COURSE_REQUIREMENT_FULL_COMBO: [2, 3, 4],
+                },
+            },
+            {
+                'id': 25,
+                'name': 'TAG生誕祭2014 俺の記録を抜いてみろ！~ HARD編 ~',
+                'level': 8,
+                'music': [
+                    (50000089, 2),
+                    (50000083, 2),
+                    (50000210, 2),
+                    (50000030, 2),
+                    (40000159, 2),
+                ],
+                'requirements': {
+                    self.COURSE_REQUIREMENT_SCORE: [800000, 900000, 931463],
+                },
+            },
+            {
+                'id': 26,
+                'name': '凍る世界で見る鳳凰の火の花',
+                'level': 9,
+                'music': [
+                    (30000043, 2),
+                    (10000039, 2),
+                    (20000048, 2),
+                    (50000096, 2),
+                    (20000038, 2),
+                ],
+                'requirements': {
+                    self.COURSE_REQUIREMENT_SCORE: [920000, 950000, 980000],
+                },
+            },
+            {
+                'id': 27,
+                'name': '真実の桜が乱れしとき、キルト纏いし君は修羅となる',
+                'level': 9,
+                'music': [
+                    (50000113, 2),
+                    (50000184, 2),
+                    (50000177, 2),
+                    (30000124, 2),
+                    (50000078, 2),
+                ],
+                'requirements': {
+                    self.COURSE_REQUIREMENT_PERFECT_PERCENT: [80, 85, 90],
+                },
+            },
+            {
+                'id': 28,
+                'name': 'THE FINAL01 ~ 雷光に月、乙女に花散る祝福を ~',
+                'level': 10,
+                'music': [
+                    (10000038, 2),
+                    (20000051, 2),
+                    (30000048, 2),
+                    (40000060, 2),
+                    (50000023, 2),
+                ],
+                'requirements': {
+                    self.COURSE_REQUIREMENT_SCORE: [920000, 950000, 980000],
+                },
+            },
+            {
+                'id': 29,
+                'name': 'The Memorial Third: assimilated all into Nature',
+                'level': 10,
+                'music': [
+                    (50000135, 2),
+                    (50000029, 2),
+                    (40000047, 2),
+                    (40000046, 2),
+                    (50000253, 2),
+                ],
+                'requirements': {
+                    self.COURSE_REQUIREMENT_SCORE: [920000, 950000, 980000],
+                },
+            },
+            {
+                'id': 30,
+                'name': '4thKAC ~ Memories of saucer ~',
+                'level': 10,
+                'music': [
+                    (50000206, 2),
+                    (50000023, 2),
+                    (50000078, 2),
+                    (50000203, 2),
+                    (50000323, 2),
+                ],
+                'requirements': {
+                    self.COURSE_REQUIREMENT_SCORE: [920000, 950000, 980000],
+                },
+            },
+        ]
+
+    def save_course(
+        self,
+        userid: UserID,
+        courseid: int,
+        rating: int,
+        scores: List[int],
+    ) -> None:
+        if len(scores) != 5:
+            raise Exception('Invalid course scores list!')
+        if rating not in [
+            self.COURSE_RATING_FAILED,
+            self.COURSE_RATING_BRONZE,
+            self.COURSE_RATING_SILVER,
+            self.COURSE_RATING_GOLD,
+        ]:
+            raise Exception('Invalid course rating!')
+
+        # Figure out if we should update the rating/scores or not
+        oldcourse = self.data.local.game.get_achievement(
+            self.game,
+            userid,
+            courseid,
+            'course',
+        )
+
+        if oldcourse is not None:
+            # Update the rating if the user did better
+            rating = max(rating, oldcourse.get_int('rating'))
+
+            # Update the scores if the total score was better
+            if sum(scores) < sum(oldcourse.get_int_array('scores', 5)):
+                scores = oldcourse.get_int_array('scores', 5)
+
+        # Save it as an achievement
+        self.data.local.game.put_achievement(
+            self.game,
+            userid,
+            courseid,
+            'course',
+            {
+                'rating': rating,
+                'scores': scores,
+            },
+        )
+
+    def get_courses(
+        self,
+        userid: UserID,
+    ) -> Dict[int, Dict[str, Any]]:
+        courses = {}
+        achievements = self.data.local.game.get_achievements(self.game, userid)
+        for achievement in achievements:
+            if achievement.type == 'course':
+                courses[achievement.id] = {
+                    'rating': achievement.data.get_int('rating'),
+                    'scores': achievement.data.get_int_array('scores', 5),
+                }
+        return courses

bemani/backend/jubeat/factory.py

@@ -0,0 +1,80 @@
+from typing import Dict, Optional, Any
+
+from bemani.backend.base import Base, Factory
+from bemani.backend.jubeat.stubs import (
+    Jubeat,
+    JubeatRipples,
+    JubeatRipplesAppend,
+    JubeatKnit,
+    JubeatKnitAppend,
+    JubeatCopious,
+    JubeatCopiousAppend,
+)
+from bemani.backend.jubeat.saucer import JubeatSaucer
+from bemani.backend.jubeat.saucerfulfill import JubeatSaucerFulfill
+from bemani.backend.jubeat.prop import JubeatProp
+from bemani.backend.jubeat.qubell import JubeatQubell
+from bemani.backend.jubeat.clan import JubeatClan
+from bemani.backend.jubeat.festo import JubeatFesto
+from bemani.common import Model
+from bemani.data import Data
+
+
+class JubeatFactory(Factory):
+
+    MANAGED_CLASSES = [
+        Jubeat,
+        JubeatRipples,
+        JubeatRipplesAppend,
+        JubeatKnit,
+        JubeatKnitAppend,
+        JubeatCopious,
+        JubeatCopiousAppend,
+        JubeatSaucer,
+        JubeatSaucerFulfill,
+        JubeatProp,
+        JubeatQubell,
+        JubeatClan,
+        JubeatFesto,
+    ]
+
+    @classmethod
+    def register_all(cls) -> None:
+        for game in ['H44', 'I44', 'J44', 'K44', 'L44']:
+            Base.register(game, JubeatFactory)
+
+    @classmethod
+    def create(cls, data: Data, config: Dict[str, Any], model: Model, parentmodel: Optional[Model]=None) -> Optional[Base]:
+        if model.game == 'H44':
+            return Jubeat(data, config, model)
+        if model.game == 'I44':
+            if model.version >= 2010031800:
+                return JubeatRipplesAppend(data, config, model)
+            else:
+                return JubeatRipples(data, config, model)
+        if model.game == 'J44':
+            if model.version >= 2011032300:
+                return JubeatKnitAppend(data, config, model)
+            else:
+                return JubeatKnit(data, config, model)
+        if model.game == 'K44':
+            if model.version >= 2012031400:
+                return JubeatCopiousAppend(data, config, model)
+            else:
+                return JubeatCopious(data, config, model)
+        if model.game == 'L44':
+            if model.version <= 2014022400:
+                return JubeatSaucer(data, config, model)
+            if model.version >= 2014030300 and model.version < 2015022000:
+                return JubeatSaucerFulfill(data, config, model)
+            if model.version >= 2015022000 and model.version < 2016033000:
+                return JubeatProp(data, config, model)
+            if model.version >= 2016033000 and model.version < 2017062600:
+                return JubeatQubell(data, config, model)
+            if model.version >= 2017062600 and model.version < 2018090500:
+                return JubeatClan(data, config, model)
+            if model.version >= 2018090500:
+                return JubeatFesto(data, config, model)
+
+        # Unknown game version
+        return None

bemani/backend/jubeat/festo.py

@@ -0,0 +1,16 @@
+# vim: set fileencoding=utf-8
+from typing import Optional
+
+from bemani.backend.jubeat.base import JubeatBase
+from bemani.backend.jubeat.clan import JubeatClan
+
+from bemani.common import VersionConstants
+
+
+class JubeatFesto(JubeatBase):
+
+    name = 'Jubeat Festo'
+    version = VersionConstants.JUBEAT_FESTO
+
+    def previous_version(self) -> Optional[JubeatBase]:
+        return JubeatClan(self.data, self.config, self.model)

bemani/backend/jubeat/saucer.py

@@ -0,0 +1,688 @@
+# vim: set fileencoding=utf-8
+import copy
+import random
+from typing import Any, Dict, List, Optional, Tuple
+
+from bemani.backend.base import Status
+from bemani.backend.jubeat.base import JubeatBase
+from bemani.backend.jubeat.common import (
+    JubeatDemodataGetHitchartHandler,
+    JubeatDemodataGetNewsHandler,
+    JubeatGamendRegisterHandler,
+    JubeatGametopGetMeetingHandler,
+    JubeatLobbyCheckHandler,
+    JubeatLoggerReportHandler,
+)
+from bemani.backend.jubeat.stubs import JubeatCopiousAppend
+from bemani.common import ValidatedDict, VersionConstants, Time
+from bemani.data import Data, Score, UserID
+from bemani.protocol import Node
+
+
+class JubeatSaucer(
+    JubeatDemodataGetHitchartHandler,
+    JubeatDemodataGetNewsHandler,
+    JubeatGamendRegisterHandler,
+    JubeatGametopGetMeetingHandler,
+    JubeatLobbyCheckHandler,
+    JubeatLoggerReportHandler,
+    JubeatBase,
+):
+
+    name = 'Jubeat Saucer'
+    version = VersionConstants.JUBEAT_SAUCER
+
+    def previous_version(self) -> Optional[JubeatBase]:
+        return JubeatCopiousAppend(self.data, self.config, self.model)
+
+    @classmethod
+    def run_scheduled_work(cls, data: Data, config: Dict[str, Any]) -> List[Tuple[str, Dict[str, Any]]]:
+        """
+        Insert daily FC challenges into the DB.
+        """
+        events = []
+        if data.local.network.should_schedule(cls.game, cls.version, 'fc_challenge', 'daily'):
+            # Generate a new list of two FC challenge songs.
+            start_time, end_time = data.local.network.get_schedule_duration('daily')
+            all_songs = set(song.id for song in data.local.music.get_all_songs(cls.game, cls.version))
+            today_song = random.sample(all_songs, 1)[0]
+            data.local.game.put_time_sensitive_settings(
+                cls.game,
+                cls.version,
+                'fc_challenge',
+                {
+                    'start_time': start_time,
+                    'end_time': end_time,
+                    'today': today_song,
+                },
+            )
+            events.append((
+                'jubeat_fc_challenge_charts',
+                {
+                    'version': cls.version,
+                    'today': today_song,
+                },
+            ))
+
+            # Mark that we did some actual work here.
+            data.local.network.mark_scheduled(cls.game, cls.version, 'fc_challenge', 'daily')
+        return events
+
+    def handle_shopinfo_regist_request(self, request: Node) -> Node:
+        # Update the name of this cab for admin purposes
+        self.update_machine_name(request.child_value('shop/name'))
+
+        shopinfo = Node.void('shopinfo')
+
+        data = Node.void('data')
+        shopinfo.add_child(data)
+        data.add_child(Node.u32('cabid', 1))
+        data.add_child(Node.string('locationid', 'nowhere'))
+        data.add_child(Node.u8('is_send', 1))
+        data.add_child(Node.s32_array(
+            'white_music_list',
+            [
+                -1, -1, -1, -1,
+                -1, -1, -1, -1,
+                -1, -1, -1, -1,
+                -1, -1, -1, -1,
+                -16385, -1, -1,
+                -1, -1, -1, -1,
+                -1, -1, -1, -1,
+                -1, -1, -1, -1,
+                -1,
+            ],
+        ))
+        data.add_child(Node.u8('tax_phase', 1))
+
+        lab = Node.void('lab')
+        data.add_child(lab)
+        lab.add_child(Node.bool('is_open', False))
+
+        vocaloid_event = Node.void('vocaloid_event')
+        data.add_child(vocaloid_event)
+        vocaloid_event.add_child(Node.u8('state', 0))
+        vocaloid_event.add_child(Node.s32('music_id', 0))
+
+        matching_off = Node.void('matching_off')
+        data.add_child(matching_off)
+        matching_off.add_child(Node.bool('is_open', True))
+
+        return shopinfo
+
+    def handle_gametop_regist_request(self, request: Node) -> Node:
+        data = request.child('data')
+        player = data.child('player')
+        passnode = player.child('pass')
+        refid = passnode.child_value('refid')
+        name = player.child_value('name')
+        root = self.new_profile_by_refid(refid, name)
+        return root
+
+    def handle_gametop_get_pdata_request(self, request: Node) -> Node:
+        data = request.child('data')
+        player = data.child('player')
+        passnode = player.child('pass')
+        refid = passnode.child_value('refid')
+        root = self.get_profile_by_refid(refid)
+        if root is None:
+            root = Node.void('gametop')
+            root.set_attribute('status', str(Status.NO_PROFILE))
+        return root
+
+    def handle_gametop_get_mdata_request(self, request: Node) -> Node:
+        data = request.child('data')
+        player = data.child('player')
+        extid = player.child_value('jid')
+        root = self.get_scores_by_extid(extid)
+        if root is None:
+            root = Node.void('gametop')
+            root.set_attribute('status', str(Status.NO_PROFILE))
+        return root
+
+    def format_profile(self, userid: UserID, profile: ValidatedDict) -> Node:
+        root = Node.void('gametop')
+        data = Node.void('data')
+        root.add_child(data)
+        player = Node.void('player')
+        data.add_child(player)
+
+        # Player info and statistics
+        info = Node.void('info')
+        player.add_child(info)
+        info.add_child(Node.s16('jubility', profile.get_int('jubility')))
+        info.add_child(Node.s16('jubility_yday', profile.get_int('jubility_yday')))
+        info.add_child(Node.s32('tune_cnt', profile.get_int('tune_cnt')))
+        info.add_child(Node.s32('save_cnt', profile.get_int('save_cnt')))
+        info.add_child(Node.s32('saved_cnt', profile.get_int('saved_cnt')))
+        info.add_child(Node.s32('fc_cnt', profile.get_int('fc_cnt')))
+        info.add_child(Node.s32('ex_cnt', profile.get_int('ex_cnt')))
+        info.add_child(Node.s32('pf_cnt', profile.get_int('pf_cnt')))
+        info.add_child(Node.s32('clear_cnt', profile.get_int('clear_cnt')))
+        info.add_child(Node.s32('match_cnt', profile.get_int('match_cnt')))
+        info.add_child(Node.s32('beat_cnt', profile.get_int('beat_cnt')))
+        info.add_child(Node.s32('mynews_cnt', profile.get_int('mynews_cnt')))
+        if 'total_best_score' in profile:
+            info.add_child(Node.s32('total_best_score', profile.get_int('total_best_score')))
+
+        # Looks to be set to true when there's an old profile, stops tutorial from
+        # happening on first load.
+        info.add_child(Node.bool('inherit', profile.get_bool('has_old_version')))
+
+        # Not saved, but loaded
+        info.add_child(Node.s32('mtg_entry_cnt', 123))
+        info.add_child(Node.s32('mtg_hold_cnt', 456))
+        info.add_child(Node.u8('mtg_result', 10))
+
+        # Secret unlocks
+        item = Node.void('item')
+        player.add_child(item)
+        item.add_child(Node.s32_array(
+            'secret_list',
+            profile.get_int_array(
+                'secret_list',
+                32,
+                [-1] * 32,
+            ),
+        ))
+        item.add_child(Node.s32_array(
+            'title_list',
+            profile.get_int_array(
+                'title_list',
+                96,
+                [-1] * 96,
+            ),
+        ))
+        item.add_child(Node.s16('theme_list', profile.get_int('theme_list', -1)))
+        item.add_child(Node.s32_array('marker_list', profile.get_int_array('marker_list', 2, [-1] * 2)))
+        item.add_child(Node.s32_array('parts_list', profile.get_int_array('parts_list', 96, [-1] * 96)))
+
+        new = Node.void('new')
+        item.add_child(new)
+        new.add_child(Node.s32_array(
+            'secret_list',
+            profile.get_int_array(
+                'secret_list_new',
+                32,
+                [-1] * 32,
+            ),
+        ))
+        new.add_child(Node.s32_array(
+            'title_list',
+            profile.get_int_array(
+                'title_list_new',
+                96,
+                [-1] * 96,
+            ),
+        ))
+        new.add_child(Node.s16('theme_list', profile.get_int('theme_list_new', -1)))
+        new.add_child(Node.s32_array('marker_list', profile.get_int_array('marker_list_new', 2, [-1] * 2)))
+
+        # Last played data, for showing cursor and such
+        lastdict = profile.get_dict('last')
+        last = Node.void('last')
+        player.add_child(last)
+        last.add_child(Node.s32('music_id', lastdict.get_int('music_id')))
+        last.add_child(Node.s8('marker', lastdict.get_int('marker')))
+        last.add_child(Node.s16('title', lastdict.get_int('title')))
+        last.add_child(Node.s8('theme', lastdict.get_int('theme')))
+        last.add_child(Node.s8('sort', lastdict.get_int('sort')))
+        last.add_child(Node.s8('rank_sort', lastdict.get_int('rank_sort')))
+        last.add_child(Node.s8('combo_disp', lastdict.get_int('combo_disp')))
+        last.add_child(Node.s8('seq_id', lastdict.get_int('seq_id')))
+        last.add_child(Node.s16('parts', lastdict.get_int('parts')))
+        last.add_child(Node.s8('category', lastdict.get_int('category')))
+        last.add_child(Node.s64('play_time', lastdict.get_int('play_time')))
+        last.add_child(Node.string('shopname', lastdict.get_str('shopname')))
+        last.add_child(Node.string('areaname', lastdict.get_str('areaname')))
+
+        # Miscelaneous crap
+        player.add_child(Node.s32('session_id', 1))
+
+        # Maybe hook this up? Unsure what it does, is it like IIDX dailies?
+        today_music = Node.void('today_music')
+        player.add_child(today_music)
+        today_music.add_child(Node.s32('music_id', 0))
+
+        # No news, ever.
+        news = Node.void('news')
+        player.add_child(news)
+        news.add_child(Node.s16('checked', 0))
+
+        # No rival support, yet.
+        rivallist = Node.void('rivallist')
+        player.add_child(rivallist)
+        rivallist.set_attribute('count', '0')
+        mylist = Node.void('mylist')
+        player.add_child(mylist)
+        mylist.set_attribute('count', '0')
+
+        # No collaboration support yet.
+        collabo = Node.void('collabo')
+        player.add_child(collabo)
+        collabo.add_child(Node.bool('success', False))
+        collabo.add_child(Node.bool('completed', False))
+
+        # Daily FC challenge.
+        entry = self.data.local.game.get_time_sensitive_settings(self.game, self.version, 'fc_challenge')
+        if entry is None:
+            entry = ValidatedDict()
+
+        # Figure out if we've played these songs
+        start_time, end_time = self.data.local.network.get_schedule_duration('daily')
+        today_attempts = self.data.local.music.get_all_attempts(self.game, self.version, userid, entry.get_int('today', -1), timelimit=start_time)
+
+        challenge = Node.void('challenge')
+        player.add_child(challenge)
+        today = Node.void('today')
+        challenge.add_child(today)
+        today.add_child(Node.s32('music_id', entry.get_int('today', -1)))
+        today.add_child(Node.u8('state', 0x40 if len(today_attempts) > 0 else 0x0))
+        onlynow = Node.void('onlynow')
+        challenge.add_child(onlynow)
+        onlynow.add_child(Node.s32('magic_no', 0))
+        onlynow.add_child(Node.s16('cycle', 0))
+
+        # Bistro event
+        bistro = Node.void('bistro')
+        player.add_child(bistro)
+
+        # Presumably these can affect the speed of the event
+        info_1 = Node.void('info')
+        bistro.add_child(info_1)
+        info_1.add_child(Node.float('delicious_rate', 1.0))
+        info_1.add_child(Node.float('favorite_rate', 1.0))
+        bistro.add_child(Node.s32('carry_over', profile.get_int('bistro_carry_over')))
+
+        # Your chef dude, I guess?
+        chefdict = profile.get_dict('chef')
+        chef = Node.void('chef')
+        bistro.add_child(chef)
+        chef.add_child(Node.s32('id', chefdict.get_int('id', 1)))
+        chef.add_child(Node.u8('ability', chefdict.get_int('ability', 2)))
+        chef.add_child(Node.u8('remain', chefdict.get_int('remain', 30)))
+        chef.add_child(Node.u8('rate', chefdict.get_int('rate', 1)))
+
+        # Routes, similar to story mode in Pop'n I guess?
+        routes = [
+            {
+                'id': 50000284,
+                'price': 20,
+                'satisfaction': 10,
+                'favorite': True,
+            },
+            {
+                'id': 50000283,
+                'price': 20,
+                'satisfaction': 20,
+                'favorite': False,
+            },
+            {
+                'id': 50000282,
+                'price': 30,
+                'satisfaction': 10,
+                'favorite': False,
+            },
+            {
+                'id': 50000275,
+                'price': 10,
+                'satisfaction': 55,
+                'favorite': False,
+            },
+            {
+                'id': 50000274,
+                'price': 40,
+                'satisfaction': 40,
+                'favorite': False,
+            },
+            {
+                'id': 50000273,
+                'price': 80,
+                'satisfaction': 60,
+                'favorite': False,
+            },
+            {
+                'id': 50000272,
+                'price': 70,
+                'satisfaction': 60,
+                'favorite': False,
+            },
+            {
+                'id': 50000271,
+                'price': 90,
+                'satisfaction': 80,
+                'favorite': False,
+            },
+            {
+                'id': 50000270,
+                'price': 90,
+                'satisfaction': 20,
+                'favorite': False,
+            },
+        ]
+        for route_no in range(len(routes)):
+            routedata = routes[route_no]
+            route = Node.void('route')
+            bistro.add_child(route)
+            route.set_attribute('no', str(route_no))
+
+            music = Node.void('music')
+            route.add_child(music)
+            music.add_child(Node.s32('id', routedata['id']))
+            music.add_child(Node.u16('price', routedata['price']))
+            music.add_child(Node.s32('price_s32', routedata['price']))
+
+            # Look up any updated satisfaction stored by the game
+            routesaved = self.data.local.user.get_achievement(self.game, self.version, userid, route_no + 1, 'route')
+            if routesaved is None:
+                routesaved = ValidatedDict()
+            satisfaction = routesaved.get_int('satisfaction', routedata['satisfaction'])
+
+            gourmates = Node.void('gourmates')
+            route.add_child(gourmates)
+            gourmates.add_child(Node.s32('id', route_no + 1))
+            gourmates.add_child(Node.u8('favorite', 1 if routedata['favorite'] else 0))
+            gourmates.add_child(Node.u16('satisfaction', satisfaction))
+            gourmates.add_child(Node.s32('satisfaction_s32', satisfaction))
+
+        # Sane defaults for unknown nodes
+        only_now_music = Node.void('only_now_music')
+        player.add_child(only_now_music)
+        only_now_music.set_attribute('count', '0')
+        requested_music = Node.void('requested_music')
+        player.add_child(requested_music)
+        requested_music.set_attribute('count', '0')
+        kac_music = Node.void('kac_music')
+        player.add_child(kac_music)
+        kac_music.set_attribute('count', '0')
+        history = Node.void('history')
+        player.add_child(history)
+        history.set_attribute('count', '0')
+
+        # Basic profile info
+        player.add_child(Node.string('name', profile.get_str('name', 'なし')))
+        player.add_child(Node.s32('jid', profile.get_int('extid')))
+        player.add_child(Node.string('refid', profile.get_str('refid')))
+
+        # Miscelaneous history stuff
+        data.add_child(Node.u8('termver', 16))
+        data.add_child(Node.u32('season_etime', 0))
+        data.add_child(Node.s32('bistro_last_music_id', 0))
+        data.add_child(Node.s32_array(
+            'white_music_list',
+            [
+                -1, -1, -1, -1, -1, -1, -1, -1,
+                -1, -1, -1, -1, -1, -1, -1, -1,
+                -1, -1, -1, -1, -1, -1, -1, -1,
+                -1, -1, -1, -1, -1, -1, -1, -1,
+            ],
+        ))
+        data.add_child(Node.s32_array(
+            'old_music_list',
+            [
+                -1, -1, -1, -1, -1, -1, -1, -1,
+                -1, -1, -1, -1, -1, -1, -1, -1,
+                -1, -1, -1, -1, -1, -1, -1, -1,
+                -1, -1, -1, -1, -1, -1, -1, -1,
+            ],
+        ))
+        data.add_child(Node.s32_array(
+            'open_music_list',
+            [
+                -1, -1, -1, -1, -1, -1, -1, -1,
+                -1, -1, -1, -1, -1, -1, -1, -1,
+                -1, -1, -1, -1, -1, -1, -1, -1,
+                -1, -1, -1, -1, -1, -1, -1, -1,
+            ],
+        ))
+
+        # Unsupported collaboration events with other games
+        collabo_info = Node.void('collabo_info')
+        data.add_child(collabo_info)
+
+        # Unsupported marathon stuff
+        run_run_marathon = Node.void('run_run_marathon')
+        collabo_info.add_child(run_run_marathon)
+        run_run_marathon.set_attribute('type', '1')
+        run_run_marathon.add_child(Node.u8('state', 1))
+        run_run_marathon.add_child(Node.bool('is_report_end', True))
+
+        # Unsupported policy break stuff
+        policy_break = Node.void('policy_break')
+        collabo_info.add_child(policy_break)
+        policy_break.set_attribute('type', '1')
+        policy_break.add_child(Node.u8('state', 1))
+        policy_break.add_child(Node.bool('is_report_end', False))
+
+        # Unsupported vocaloid stuff
+        vocaloid_event = Node.void('vocaloid_event')
+        collabo_info.add_child(vocaloid_event)
+        vocaloid_event.set_attribute('type', '1')
+        vocaloid_event.add_child(Node.u8('state', 0))
+        vocaloid_event.add_child(Node.s32('music_id', 0))
+
+        # No obnoxious 30 second wait to play.
+        matching_off = Node.void('matching_off')
+        data.add_child(matching_off)
+        matching_off.add_child(Node.bool('is_open', True))
+
+        return root
+
+    def unformat_profile(self, userid: UserID, request: Node, oldprofile: ValidatedDict) -> ValidatedDict:
+        newprofile = copy.deepcopy(oldprofile)
+        data = request.child('data')
+
+        # Grab player information
+        player = data.child('player')
+
+        # Grab last information. Lots of this will be filled in while grabbing scores
+        last = newprofile.get_dict('last')
+        last.replace_int('play_time', player.child_value('time_gameend'))
+        last.replace_str('shopname', player.child_value('shopname'))
+        last.replace_str('areaname', player.child_value('areaname'))
+
+        # Grab player info for echoing back
+        info = player.child('info')
+        if info is not None:
+            newprofile.replace_int('jubility', info.child_value('jubility'))
+            newprofile.replace_int('jubility_yday', info.child_value('jubility_yday'))
+            newprofile.replace_int('tune_cnt', info.child_value('tune_cnt'))
+            newprofile.replace_int('save_cnt', info.child_value('save_cnt'))
+            newprofile.replace_int('saved_cnt', info.child_value('saved_cnt'))
+            newprofile.replace_int('fc_cnt', info.child_value('fc_cnt'))
+            newprofile.replace_int('ex_cnt', info.child_value('exc_cnt'))  # Not a mistake, Jubeat is weird
+            newprofile.replace_int('pf_cnt', info.child_value('pf_cnt'))
+            newprofile.replace_int('clear_cnt', info.child_value('clear_cnt'))
+            newprofile.replace_int('match_cnt', info.child_value('match_cnt'))
+            newprofile.replace_int('beat_cnt', info.child_value('beat_cnt'))
+            newprofile.replace_int('total_best_score', info.child_value('total_best_score'))
+            newprofile.replace_int('mynews_cnt', info.child_value('mynews_cnt'))
+
+        # Grab unlock progress
+        item = player.child('item')
+        if item is not None:
+            newprofile.replace_int_array('secret_list', 32, item.child_value('secret_list'))
+            newprofile.replace_int_array('title_list', 96, item.child_value('title_list'))
+            newprofile.replace_int('theme_list', item.child_value('theme_list'))
+            newprofile.replace_int_array('marker_list', 2, item.child_value('marker_list'))
+            newprofile.replace_int_array('parts_list', 96, item.child_value('parts_list'))
+            newprofile.replace_int_array('secret_list_new', 32, item.child_value('secret_new'))
+            newprofile.replace_int_array('title_list_new', 96, item.child_value('title_new'))
+            newprofile.replace_int('theme_list_new', item.child_value('theme_new'))
+            newprofile.replace_int_array('marker_list_new', 2, item.child_value('marker_new'))
+
+        # Grab bistro progress
+        bistro = player.child('bistro')
+        if bistro is not None:
+            newprofile.replace_int('bistro_carry_over', bistro.child_value('carry_over'))
+
+            chefdata = newprofile.get_dict('chef')
+            chef = bistro.child('chef')
+            if chef is not None:
+                chefdata.replace_int('id', chef.child_value('id'))
+                chefdata.replace_int('ability', chef.child_value('ability'))
+                chefdata.replace_int('remain', chef.child_value('remain'))
+                chefdata.replace_int('rate', chef.child_value('rate'))
+            newprofile.replace_dict('chef', chefdata)
+
+            for route in bistro.children:
+                if route.name != 'route':
+                    continue
+
+                gourmates = route.child('gourmates')
+                routeid = gourmates.child_value('id')
+                satisfaction = gourmates.child_value('satisfaction_s32')
+                self.data.local.user.put_achievement(
+                    self.game,
+                    self.version,
+                    userid,
+                    routeid,
+                    'route',
+                    {
+                        'satisfaction': satisfaction,
+                    },
+                )
+
+        # Get timestamps for played songs
+        timestamps: Dict[int, int] = {}
+        history = player.child('history')
+        if history is not None:
+            for tune in history.children:
+                if tune.name != 'tune':
+                    continue
+                entry = int(tune.attribute('log_id'))
+                ts = int(tune.child_value('timestamp') / 1000)
+                timestamps[entry] = ts
+
+        # Grab scores and save those
+        result = data.child('result')
+        if result is not None:
+            for tune in result.children:
+                if tune.name != 'tune':
+                    continue
+                result = tune.child('player')
+
+                last.replace_int('marker', tune.child_value('marker'))
+                last.replace_int('title', tune.child_value('title'))
+                last.replace_int('parts', tune.child_value('parts'))
+                last.replace_int('theme', tune.child_value('theme'))
+                last.replace_int('sort', tune.child_value('sort'))
+                last.replace_int('category', tune.child_value('category'))
+                last.replace_int('rank_sort', tune.child_value('rank_sort'))
+                last.replace_int('combo_disp', tune.child_value('combo_disp'))
+
+                songid = tune.child_value('music')
+                entry = int(tune.attribute('id'))
+                timestamp = timestamps.get(entry, Time.now())
+                chart = int(result.child('score').attribute('seq'))
+                points = result.child_value('score')
+                flags = int(result.child('score').attribute('clear'))
+                combo = int(result.child('score').attribute('combo'))
+                ghost = result.child_value('mbar')
+
+                # Miscelaneous last data for echoing to profile get
+                last.replace_int('music_id', songid)
+                last.replace_int('seq_id', chart)
+
+                mapping = {
+                    self.GAME_FLAG_BIT_CLEARED: self.PLAY_MEDAL_CLEARED,
+                    self.GAME_FLAG_BIT_FULL_COMBO: self.PLAY_MEDAL_FULL_COMBO,
+                    self.GAME_FLAG_BIT_EXCELLENT: self.PLAY_MEDAL_EXCELLENT,
+                    self.GAME_FLAG_BIT_NEARLY_FULL_COMBO: self.PLAY_MEDAL_NEARLY_FULL_COMBO,
+                    self.GAME_FLAG_BIT_NEARLY_EXCELLENT: self.PLAY_MEDAL_NEARLY_EXCELLENT,
+                }
+
+                # Figure out the highest medal based on bits passed in
+                medal = self.PLAY_MEDAL_FAILED
+                for bit in mapping:
+                    if flags & bit > 0:
+                        medal = max(medal, mapping[bit])
+
+                self.update_score(userid, timestamp, songid, chart, points, medal, combo, ghost)
+
+        # Save back last information gleaned from results
+        newprofile.replace_dict('last', last)
+
+        # Keep track of play statistics
+        self.update_play_statistics(userid)
+
+        return newprofile
+
+    def format_scores(self, userid: UserID, profile: ValidatedDict, scores: List[Score]) -> Node:
+        scores = self.data.remote.music.get_scores(self.game, self.version, userid)
+
+        root = Node.void('gametop')
+        datanode = Node.void('data')
+        root.add_child(datanode)
+        player = Node.void('player')
+        datanode.add_child(player)
+        playdata = Node.void('playdata')
+        player.add_child(playdata)
+        playdata.set_attribute('count', str(len(scores)))
+
+        music = ValidatedDict()
+        for score in scores:
+            data = music.get_dict(str(score.id))
+            play_cnt = data.get_int_array('play_cnt', 3)
+            clear_cnt = data.get_int_array('clear_cnt', 3)
+            clear_flags = data.get_int_array('clear_flags', 3)
+            fc_cnt = data.get_int_array('fc_cnt', 3)
+            ex_cnt = data.get_int_array('ex_cnt', 3)
+            points = data.get_int_array('points', 3)
+
+            # Replace data for this chart type
+            play_cnt[score.chart] = score.plays
+            clear_cnt[score.chart] = score.data.get_int('clear_count')
+            fc_cnt[score.chart] = score.data.get_int('full_combo_count')
+            ex_cnt[score.chart] = score.data.get_int('excellent_count')
+            points[score.chart] = score.points
+
+            # Format the clear flags
+            clear_flags[score.chart] = self.GAME_FLAG_BIT_PLAYED
+            if score.data.get_int('clear_count') > 0:
+                clear_flags[score.chart] |= self.GAME_FLAG_BIT_CLEARED
+            if score.data.get_int('full_combo_count') > 0:
+                clear_flags[score.chart] |= self.GAME_FLAG_BIT_FULL_COMBO
+            if score.data.get_int('excellent_count') > 0:
+                clear_flags[score.chart] |= self.GAME_FLAG_BIT_EXCELLENT
+
+            # Save chart data back
+            data.replace_int_array('play_cnt', 3, play_cnt)
+            data.replace_int_array('clear_cnt', 3, clear_cnt)
+            data.replace_int_array('clear_flags', 3, clear_flags)
+            data.replace_int_array('fc_cnt', 3, fc_cnt)
+            data.replace_int_array('ex_cnt', 3, ex_cnt)
+            data.replace_int_array('points', 3, points)
+
+            # Update the ghost (untyped)
+            ghost = data.get('ghost', [None, None, None])
+            ghost[score.chart] = score.data.get('ghost')
+            data['ghost'] = ghost
+
+            # Save it back
+            music.replace_dict(str(score.id), data)
+
+        for scoreid in music:
+            scoredata = music[scoreid]
+            musicdata = Node.void('musicdata')
+            playdata.add_child(musicdata)
+
+            musicdata.set_attribute('music_id', scoreid)
+            musicdata.add_child(Node.s32_array('play_cnt', scoredata.get_int_array('play_cnt', 3)))
+            musicdata.add_child(Node.s32_array('clear_cnt', scoredata.get_int_array('clear_cnt', 3)))
+            musicdata.add_child(Node.s32_array('fc_cnt', scoredata.get_int_array('fc_cnt', 3)))
+            musicdata.add_child(Node.s32_array('ex_cnt', scoredata.get_int_array('ex_cnt', 3)))
+            musicdata.add_child(Node.s32_array('score', scoredata.get_int_array('points', 3)))
+            musicdata.add_child(Node.s8_array('clear', scoredata.get_int_array('clear_flags', 3)))
+
+            ghosts = scoredata.get('ghost', [None, None, None])
+            for i in range(len(ghosts)):
+                ghost = ghosts[i]
+                if ghost is None:
+                    continue
+
+                bar = Node.u8_array('bar', ghost)
+                musicdata.add_child(bar)
+                bar.set_attribute('seq', str(i))
+
+        return root

bemani/backend/jubeat/saucerfulfill.py

@@ -0,0 +1,787 @@
+# vim: set fileencoding=utf-8
+import copy
+import random
+from typing import Any, Dict, List, Optional, Tuple
+
+from bemani.backend.base import Status
+from bemani.backend.jubeat.base import JubeatBase
+from bemani.backend.jubeat.common import (
+    JubeatDemodataGetHitchartHandler,
+    JubeatDemodataGetNewsHandler,
+    JubeatGamendRegisterHandler,
+    JubeatGametopGetMeetingHandler,
+    JubeatLobbyCheckHandler,
+    JubeatLoggerReportHandler,
+)
+from bemani.backend.jubeat.course import JubeatCourse
+from bemani.backend.jubeat.saucer import JubeatSaucer
+from bemani.common import ValidatedDict, VersionConstants, Time
+from bemani.data import Data, Score, UserID
+from bemani.protocol import Node
+
+
+class JubeatSaucerFulfill(
+    JubeatDemodataGetHitchartHandler,
+    JubeatDemodataGetNewsHandler,
+    JubeatGamendRegisterHandler,
+    JubeatGametopGetMeetingHandler,
+    JubeatLobbyCheckHandler,
+    JubeatLoggerReportHandler,
+    JubeatCourse,
+    JubeatBase,
+):
+
+    name = 'Jubeat Saucer Fulfill'
+    version = VersionConstants.JUBEAT_SAUCER_FULFILL
+
+    GAME_COURSE_REQUIREMENT_SCORE = 1
+    GAME_COURSE_REQUIREMENT_FULL_COMBO = 2
+    GAME_COURSE_REQUIREMENT_PERFECT_PERCENT = 3
+
+    GAME_COURSE_RATING_FAILED = 1
+    GAME_COURSE_RATING_BRONZE = 2
+    GAME_COURSE_RATING_SILVER = 3
+    GAME_COURSE_RATING_GOLD = 4
+
+    def previous_version(self) -> Optional[JubeatBase]:
+        return JubeatSaucer(self.data, self.config, self.model)
+
+    @classmethod
+    def run_scheduled_work(cls, data: Data, config: Dict[str, Any]) -> List[Tuple[str, Dict[str, Any]]]:
+        """
+        Insert daily FC challenges into the DB.
+        """
+        events = []
+        if data.local.network.should_schedule(cls.game, cls.version, 'fc_challenge', 'daily'):
+            # Generate a new list of two FC challenge songs.
+            start_time, end_time = data.local.network.get_schedule_duration('daily')
+            all_songs = set(song.id for song in data.local.music.get_all_songs(cls.game, cls.version))
+            daily_songs = random.sample(all_songs, 2)
+            data.local.game.put_time_sensitive_settings(
+                cls.game,
+                cls.version,
+                'fc_challenge',
+                {
+                    'start_time': start_time,
+                    'end_time': end_time,
+                    'today': daily_songs[0],
+                    'whim': daily_songs[1],
+                },
+            )
+            events.append((
+                'jubeat_fc_challenge_charts',
+                {
+                    'version': cls.version,
+                    'today': daily_songs[0],
+                    'whim': daily_songs[1],
+                },
+            ))
+
+            # Mark that we did some actual work here.
+            data.local.network.mark_scheduled(cls.game, cls.version, 'fc_challenge', 'daily')
+        return events
+
+    def handle_shopinfo_regist_request(self, request: Node) -> Node:
+        # Update the name of this cab for admin purposes
+        self.update_machine_name(request.child_value('shop/name'))
+
+        shopinfo = Node.void('shopinfo')
+
+        data = Node.void('data')
+        shopinfo.add_child(data)
+        data.add_child(Node.u32('cabid', 1))
+        data.add_child(Node.string('locationid', 'nowhere'))
+        data.add_child(Node.u8('is_send', 1))
+        data.add_child(Node.s32_array(
+            'white_music_list',
+            [
+                -1, -1, -1, -1,
+                -1, -1, -1, -1,
+                -1, -1, -1, -1,
+                -1, -1, -1, -1,
+                -1, -1, -1, -1,
+                -1, -1, -1, -1,
+                -1, -1, -1, -1,
+                -1, -1, -1, -1,
+            ],
+        ))
+        data.add_child(Node.u8('tax_phase', 1))
+
+        lab = Node.void('lab')
+        data.add_child(lab)
+        lab.add_child(Node.bool('is_open', False))
+
+        vocaloid_event = Node.void('vocaloid_event')
+        data.add_child(vocaloid_event)
+        vocaloid_event.add_child(Node.u8('state', 0))
+        vocaloid_event.add_child(Node.s32('music_id', 0))
+
+        vocaloid_event2 = Node.void('vocaloid_event2')
+        data.add_child(vocaloid_event2)
+        vocaloid_event2.add_child(Node.u8('state', 0))
+        vocaloid_event2.add_child(Node.s32('music_id', 0))
+
+        # No obnoxious 30 second wait to play.
+        matching_off = Node.void('matching_off')
+        data.add_child(matching_off)
+        matching_off.add_child(Node.bool('is_open', True))
+
+        tenka = Node.void('tenka')
+        data.add_child(tenka)
+        tenka.add_child(Node.bool('is_participant', False))
+
+        return shopinfo
+
+    def handle_gametop_get_course_request(self, request: Node) -> Node:
+        data = request.child('data')
+        player = data.child('player')
+        extid = player.child_value('jid')
+
+        gametop = Node.void('gametop')
+        data = Node.void('data')
+        gametop.add_child(data)
+
+        # Course list available
+        course_list = Node.void('course_list')
+        data.add_child(course_list)
+
+        validcourses: List[int] = []
+        for course in self.get_all_courses():
+            coursenode = Node.void('course')
+            course_list.add_child(coursenode)
+
+            # Basic course info
+            if course['id'] in validcourses:
+                raise Exception('Cannot have same course ID specified twice!')
+            validcourses.append(course['id'])
+            coursenode.add_child(Node.s32('id', course['id']))
+            coursenode.add_child(Node.string('name', course['name']))
+            coursenode.add_child(Node.u8('level', course['level']))
+
+            # Translate internal to game
+            def translate_req(internal_req: int) -> int:
+                return {
+                    self.COURSE_REQUIREMENT_SCORE: self.GAME_COURSE_REQUIREMENT_SCORE,
+                    self.COURSE_REQUIREMENT_FULL_COMBO: self.GAME_COURSE_REQUIREMENT_FULL_COMBO,
+                    self.COURSE_REQUIREMENT_PERFECT_PERCENT: self.GAME_COURSE_REQUIREMENT_PERFECT_PERCENT,
+                }.get(internal_req, 0)
+
+            # Course bronze/silver/gold rules
+            ids = [0] * 3
+            bronze_values = [0] * 3
+            silver_values = [0] * 3
+            gold_values = [0] * 3
+            slot = 0
+            for req in course['requirements']:
+                req_values = course['requirements'][req]
+
+                ids[slot] = translate_req(req)
+                bronze_values[slot] = req_values[0]
+                silver_values[slot] = req_values[1]
+                gold_values[slot] = req_values[2]
+                slot = slot + 1
+
+            norma = Node.void('norma')
+            coursenode.add_child(norma)
+            norma.add_child(Node.s32_array('norma_id', ids))
+            norma.add_child(Node.s32_array('bronze_value', bronze_values))
+            norma.add_child(Node.s32_array('silver_value', silver_values))
+            norma.add_child(Node.s32_array('gold_value', gold_values))
+
+            # Music list for course
+            music_index = 0
+            music_list = Node.void('music_list')
+            coursenode.add_child(music_list)
+
+            for entry in course['music']:
+                music = Node.void('music')
+                music.set_attribute('index', str(music_index))
+                music_list.add_child(music)
+                music.add_child(Node.s32('music_id', entry[0]))
+                music.add_child(Node.u8('seq', entry[1]))
+                music_index = music_index + 1
+
+        # Look up profile so we can load the last course played
+        userid = self.data.remote.user.from_extid(self.game, self.version, extid)
+        profile = self.get_profile(userid)
+        if profile is None:
+            profile = ValidatedDict()
+
+        # Player scores for courses
+        player_list = Node.void('player_list')
+        data.add_child(player_list)
+        player = Node.void('player')
+        player_list.add_child(player)
+        player.add_child(Node.s32('jid', extid))
+
+        result_list = Node.void('result_list')
+        player.add_child(result_list)
+        playercourses = self.get_courses(userid)
+        for courseid in playercourses:
+            if courseid not in validcourses:
+                continue
+
+            rating = {
+                self.COURSE_RATING_FAILED: self.GAME_COURSE_RATING_FAILED,
+                self.COURSE_RATING_BRONZE: self.GAME_COURSE_RATING_BRONZE,
+                self.COURSE_RATING_SILVER: self.GAME_COURSE_RATING_SILVER,
+                self.COURSE_RATING_GOLD: self.GAME_COURSE_RATING_GOLD,
+            }[playercourses[courseid]['rating']]
+            scores = playercourses[courseid]['scores']
+
+            result = Node.void('result')
+            result_list.add_child(result)
+            result.add_child(Node.s32('id', courseid))
+            result.add_child(Node.u8('rating', rating))
+            result.add_child(Node.s32_array('score', scores))
+
+        # Last course ID
+        last_course_id = Node.s32('last_course_id', profile.get_dict('last').get_int('last_course_id', -1))
+        data.add_child(last_course_id)
+
+        return gametop
+
+    def handle_gametop_regist_request(self, request: Node) -> Node:
+        data = request.child('data')
+        player = data.child('player')
+        passnode = player.child('pass')
+        refid = passnode.child_value('refid')
+        name = player.child_value('name')
+        root = self.new_profile_by_refid(refid, name)
+        return root
+
+    def handle_gametop_get_pdata_request(self, request: Node) -> Node:
+        data = request.child('data')
+        player = data.child('player')
+        passnode = player.child('pass')
+        refid = passnode.child_value('refid')
+        root = self.get_profile_by_refid(refid)
+        if root is None:
+            root = Node.void('gametop')
+            root.set_attribute('status', str(Status.NO_PROFILE))
+        return root
+
+    def handle_gametop_get_mdata_request(self, request: Node) -> Node:
+        data = request.child('data')
+        player = data.child('player')
+        extid = player.child_value('jid')
+        root = self.get_scores_by_extid(extid)
+        if root is None:
+            root = Node.void('gametop')
+            root.set_attribute('status', str(Status.NO_PROFILE))
+        return root
+
+    def format_profile(self, userid: UserID, profile: ValidatedDict) -> Node:
+        root = Node.void('gametop')
+        data = Node.void('data')
+        root.add_child(data)
+        player = Node.void('player')
+        data.add_child(player)
+
+        # Player info and statistics
+        info = Node.void('info')
+        player.add_child(info)
+        info.add_child(Node.s16('jubility', profile.get_int('jubility')))
+        info.add_child(Node.s16('jubility_yday', profile.get_int('jubility_yday')))
+        info.add_child(Node.s32('tune_cnt', profile.get_int('tune_cnt')))
+        info.add_child(Node.s32('save_cnt', profile.get_int('save_cnt')))
+        info.add_child(Node.s32('saved_cnt', profile.get_int('saved_cnt')))
+        info.add_child(Node.s32('fc_cnt', profile.get_int('fc_cnt')))
+        info.add_child(Node.s32('ex_cnt', profile.get_int('ex_cnt')))
+        info.add_child(Node.s32('pf_cnt', profile.get_int('pf_cnt')))
+        info.add_child(Node.s32('clear_cnt', profile.get_int('clear_cnt')))
+        info.add_child(Node.s32('match_cnt', profile.get_int('match_cnt')))
+        info.add_child(Node.s32('beat_cnt', profile.get_int('beat_cnt')))
+        info.add_child(Node.s32('mynews_cnt', profile.get_int('mynews_cnt')))
+        info.add_child(Node.s32('extra_point', profile.get_int('extra_point')))
+        info.add_child(Node.bool('is_extra_played', profile.get_bool('is_extra_played')))
+        if 'total_best_score' in profile:
+            info.add_child(Node.s32('total_best_score', profile.get_int('total_best_score')))
+
+        # Looks to be set to true when there's an old profile, stops tutorial from
+        # happening on first load.
+        info.add_child(Node.bool('inherit', profile.get_bool('has_old_version')))
+
+        # Not saved, but loaded
+        info.add_child(Node.s32('mtg_entry_cnt', 123))
+        info.add_child(Node.s32('mtg_hold_cnt', 456))
+        info.add_child(Node.u8('mtg_result', 10))
+
+        # First play stuff we don't support
+        free_first_play = Node.void('free_first_play')
+        player.add_child(free_first_play)
+        free_first_play.add_child(Node.bool('is_available', False))
+        free_first_play.add_child(Node.s32('point', 0))
+        free_first_play.add_child(Node.s32('point_used', 0))
+
+        # Secret unlocks
+        item = Node.void('item')
+        player.add_child(item)
+        item.add_child(Node.s32_array(
+            'secret_list',
+            profile.get_int_array(
+                'secret_list',
+                32,
+                [-1] * 32,
+            ),
+        ))
+        item.add_child(Node.s32_array(
+            'title_list',
+            profile.get_int_array(
+                'title_list',
+                96,
+                [-1] * 96,
+            ),
+        ))
+        item.add_child(Node.s16('theme_list', profile.get_int('theme_list', -1)))
+        item.add_child(Node.s32_array('marker_list', profile.get_int_array('marker_list', 2, [-1] * 2)))
+        item.add_child(Node.s32_array('parts_list', profile.get_int_array('parts_list', 96, [-1] * 96)))
+
+        new = Node.void('new')
+        item.add_child(new)
+        new.add_child(Node.s32_array(
+            'secret_list',
+            profile.get_int_array(
+                'secret_list_new',
+                32,
+                [-1] * 32,
+            ),
+        ))
+        new.add_child(Node.s32_array(
+            'title_list',
+            profile.get_int_array(
+                'title_list_new',
+                96,
+                [-1] * 96,
+            ),
+        ))
+        new.add_child(Node.s16('theme_list', profile.get_int('theme_list_new', -1)))
+        new.add_child(Node.s32_array('marker_list', profile.get_int_array('marker_list_new', 2, [-1] * 2)))
+
+        # Last played data, for showing cursor and such
+        lastdict = profile.get_dict('last')
+        last = Node.void('last')
+        player.add_child(last)
+        last.add_child(Node.s32('music_id', lastdict.get_int('music_id')))
+        last.add_child(Node.s8('marker', lastdict.get_int('marker')))
+        last.add_child(Node.s16('title', lastdict.get_int('title')))
+        last.add_child(Node.s8('theme', lastdict.get_int('theme')))
+        last.add_child(Node.s8('sort', lastdict.get_int('sort')))
+        last.add_child(Node.s8('rank_sort', lastdict.get_int('rank_sort')))
+        last.add_child(Node.s8('combo_disp', lastdict.get_int('combo_disp')))
+        last.add_child(Node.s8('seq_id', lastdict.get_int('seq_id')))
+        last.add_child(Node.s16('parts', lastdict.get_int('parts')))
+        last.add_child(Node.s8('category', lastdict.get_int('category')))
+        last.add_child(Node.s64('play_time', lastdict.get_int('play_time')))
+        last.add_child(Node.string('shopname', lastdict.get_str('shopname')))
+        last.add_child(Node.string('areaname', lastdict.get_str('areaname')))
+        last.add_child(Node.s8('expert_option', lastdict.get_int('expert_option')))
+        last.add_child(Node.s8('matching', lastdict.get_int('matching')))
+        last.add_child(Node.s8('hazard', lastdict.get_int('hazard')))
+        last.add_child(Node.s8('hard', lastdict.get_int('hard')))
+
+        # Miscelaneous crap
+        player.add_child(Node.s32('session_id', 1))
+        player.add_child(Node.u64('event_flag', 0))
+
+        # Macchiato event
+        macchiatodict = profile.get_dict('macchiato')
+        macchiato = Node.void('macchiato')
+        player.add_child(macchiato)
+        macchiato.add_child(Node.s32('pack_id', macchiatodict.get_int('pack_id')))
+        macchiato.add_child(Node.u16('bean_num', macchiatodict.get_int('bean_num')))
+        macchiato.add_child(Node.s32('daily_milk_num', macchiatodict.get_int('daily_milk_num')))
+        macchiato.add_child(Node.bool('is_received_daily_milk', macchiatodict.get_bool('is_received_daily_milk')))
+        macchiato.add_child(Node.s32('today_tune_cnt', macchiatodict.get_int('today_tune_cnt')))
+        macchiato.add_child(Node.s32_array(
+            'daily_milk_bonus',
+            macchiatodict.get_int_array('daily_milk_bonus', 9, [-1, -1, -1, -1, -1, -1, -1, -1, -1]),
+        ))
+        macchiato.add_child(Node.s32('daily_play_burst', macchiatodict.get_int('daily_play_burst')))
+        macchiato.add_child(Node.bool('sub_menu_is_completed', macchiatodict.get_bool('sub_menu_is_completed')))
+        macchiato.add_child(Node.s32('compensation_milk', macchiatodict.get_int('compensation_milk')))
+        macchiato.add_child(Node.s32('match_cnt', macchiatodict.get_int('match_cnt')))
+
+        # Probably never will support this
+        macchiato_music_list = Node.void('macchiato_music_list')
+        macchiato.add_child(macchiato_music_list)
+        macchiato_music_list.set_attribute('count', '0')
+
+        # Same with this comment
+        macchiato.add_child(Node.s32('sub_pack_id', 0))
+        sub_macchiato_music_list = Node.void('sub_macchiato_music_list')
+        macchiato.add_child(sub_macchiato_music_list)
+        sub_macchiato_music_list.set_attribute('count', '0')
+
+        # And this
+        season_music_list = Node.void('season_music_list')
+        macchiato.add_child(season_music_list)
+        season_music_list.set_attribute('count', '0')
+
+        # Weird, this is sent as a void with a bunch of subnodes, but returned as an int array.
+        achievement_list = Node.void('achievement_list')
+        macchiato.add_child(achievement_list)
+        achievement_list.set_attribute('count', '0')
+
+        # Also probably never supporting this either
+        cow_list = Node.void('cow_list')
+        macchiato.add_child(cow_list)
+        cow_list.set_attribute('count', '0')
+
+        # No news, ever.
+        news = Node.void('news')
+        player.add_child(news)
+        news.add_child(Node.s16('checked', 0))
+
+        # No rival support, yet.
+        rivallist = Node.void('rivallist')
+        player.add_child(rivallist)
+        rivallist.set_attribute('count', '0')
+
+        # Full combo daily challenge.
+        entry = self.data.local.game.get_time_sensitive_settings(self.game, self.version, 'fc_challenge')
+        if entry is None:
+            entry = ValidatedDict()
+
+        # Figure out if we've played these songs
+        start_time, end_time = self.data.local.network.get_schedule_duration('daily')
+        today_attempts = self.data.local.music.get_all_attempts(self.game, self.version, userid, entry.get_int('today', -1), timelimit=start_time)
+        whim_attempts = self.data.local.music.get_all_attempts(self.game, self.version, userid, entry.get_int('whim', -1), timelimit=start_time)
+
+        challenge = Node.void('challenge')
+        player.add_child(challenge)
+        today = Node.void('today')
+        challenge.add_child(today)
+        today.add_child(Node.s32('music_id', entry.get_int('today', -1)))
+        today.add_child(Node.u8('state', 0x40 if len(today_attempts) > 0 else 0x0))
+        whim = Node.void('whim')
+        challenge.add_child(whim)
+        whim.add_child(Node.s32('music_id', entry.get_int('whim', -1)))
+        whim.add_child(Node.u8('state', 0x40 if len(whim_attempts) > 0 else 0x0))
+
+        # Sane defaults for unknown nodes
+        only_now_music = Node.void('only_now_music')
+        player.add_child(only_now_music)
+        only_now_music.set_attribute('count', '0')
+        lab_edit_seq = Node.void('lab_edit_seq')
+        player.add_child(lab_edit_seq)
+        lab_edit_seq.set_attribute('count', '0')
+        kac_music = Node.void('kac_music')
+        player.add_child(kac_music)
+        kac_music.set_attribute('count', '0')
+        history = Node.void('history')
+        player.add_child(history)
+        history.set_attribute('count', '0')
+        share_music = Node.void('share_music')
+        player.add_child(share_music)
+        share_music.set_attribute('count', '0')
+        bonus_music = Node.void('bonus_music')
+        player.add_child(bonus_music)
+        bonus_music.set_attribute('count', '0')
+
+        bingo = Node.void('bingo')
+        player.add_child(bingo)
+        reward = Node.void('reward')
+        bingo.add_child(reward)
+        reward.add_child(Node.s32('total', 0))
+        reward.add_child(Node.s32('point', 0))
+        group = Node.void('group')
+        player.add_child(group)
+        group.add_child(Node.s32('group_id', 0))
+
+        # Basic profile info
+        player.add_child(Node.string('name', profile.get_str('name', 'なし')))
+        player.add_child(Node.s32('jid', profile.get_int('extid')))
+        player.add_child(Node.string('refid', profile.get_str('refid')))
+
+        # Miscelaneous history stuff
+        data.add_child(Node.u8('termver', 16))
+        data.add_child(Node.u32('season_etime', 0))
+        data.add_child(Node.s32_array(
+            'white_music_list',
+            [
+                -1, -1, -1, -1, -1, -1, -1, -1,
+                -1, -1, -1, -1, -1, -1, -1, -1,
+                -1, -1, -1, -1, -1, -1, -1, -1,
+                -1, -1, -1, -1, -1, -1, -1, -1,
+            ],
+        ))
+        data.add_child(Node.s32_array(
+            'open_music_list',
+            [
+                -1, -1, -1, -1, -1, -1, -1, -1,
+                -1, -1, -1, -1, -1, -1, -1, -1,
+                -1, -1, -1, -1, -1, -1, -1, -1,
+                -1, -1, -1, -1, -1, -1, -1, -1,
+            ],
+        ))
+
+        # Unsupported collaboration events with other games
+        collabo_info = Node.void('collabo_info')
+        data.add_child(collabo_info)
+
+        # Unsupported policy break stuff
+        policy_break = Node.void('policy_break')
+        collabo_info.add_child(policy_break)
+        policy_break.set_attribute('type', '1')
+        policy_break.add_child(Node.u8('state', 1))
+        policy_break.add_child(Node.bool('is_report_end', False))
+
+        # Unsupported vocaloid stuff
+        vocaloid_event = Node.void('vocaloid_event')
+        collabo_info.add_child(vocaloid_event)
+        vocaloid_event.set_attribute('type', '1')
+        vocaloid_event.add_child(Node.u8('state', 0))
+        vocaloid_event.add_child(Node.s32('music_id', 0))
+
+        # Unsupported vocaloid stuff
+        vocaloid_event2 = Node.void('vocaloid_event2')
+        collabo_info.add_child(vocaloid_event2)
+        vocaloid_event2.set_attribute('type', '1')
+        vocaloid_event2.add_child(Node.u8('state', 0))
+        vocaloid_event2.add_child(Node.s32('music_id', 0))
+
+        # Maybe it is possible to turn off internet matching here?
+        lab = Node.void('lab')
+        data.add_child(lab)
+        lab.add_child(Node.bool('is_open', False))
+        matching_off = Node.void('matching_off')
+        data.add_child(matching_off)
+        matching_off.add_child(Node.bool('is_open', True))
+
+        return root
+
+    def unformat_profile(self, userid: UserID, request: Node, oldprofile: ValidatedDict) -> ValidatedDict:
+        newprofile = copy.deepcopy(oldprofile)
+        data = request.child('data')
+
+        # Grab player information
+        player = data.child('player')
+
+        # Grab last information. Lots of this will be filled in while grabbing scores
+        last = newprofile.get_dict('last')
+        last.replace_int('play_time', player.child_value('time_gameend'))
+        last.replace_str('shopname', player.child_value('shopname'))
+        last.replace_str('areaname', player.child_value('areaname'))
+
+        # Grab player info for echoing back
+        info = player.child('info')
+        if info is not None:
+            newprofile.replace_int('jubility', info.child_value('jubility'))
+            newprofile.replace_int('jubility_yday', info.child_value('jubility_yday'))
+            newprofile.replace_int('tune_cnt', info.child_value('tune_cnt'))
+            newprofile.replace_int('save_cnt', info.child_value('save_cnt'))
+            newprofile.replace_int('saved_cnt', info.child_value('saved_cnt'))
+            newprofile.replace_int('fc_cnt', info.child_value('fc_cnt'))
+            newprofile.replace_int('ex_cnt', info.child_value('exc_cnt'))  # Not a mistake, Jubeat is weird
+            newprofile.replace_int('pf_cnt', info.child_value('pf_cnt'))
+            newprofile.replace_int('clear_cnt', info.child_value('clear_cnt'))
+            newprofile.replace_int('match_cnt', info.child_value('match_cnt'))
+            newprofile.replace_int('beat_cnt', info.child_value('beat_cnt'))
+            newprofile.replace_int('total_best_score', info.child_value('total_best_score'))
+            newprofile.replace_int('mynews_cnt', info.child_value('mynews_cnt'))
+            newprofile.replace_int('extra_point', info.child_value('extra_point'))
+            newprofile.replace_bool('is_extra_played', info.child_value('is_extra_played'))
+
+            last.replace_int('expert_option', info.child_value('expert_option'))
+            last.replace_int('matching', info.child_value('matching'))
+            last.replace_int('hazard', info.child_value('hazard'))
+            last.replace_int('hard', info.child_value('hard'))
+
+        # Grab unlock progress
+        item = player.child('item')
+        if item is not None:
+            newprofile.replace_int_array('secret_list', 32, item.child_value('secret_list'))
+            newprofile.replace_int_array('title_list', 96, item.child_value('title_list'))
+            newprofile.replace_int('theme_list', item.child_value('theme_list'))
+            newprofile.replace_int_array('marker_list', 2, item.child_value('marker_list'))
+            newprofile.replace_int_array('parts_list', 96, item.child_value('parts_list'))
+            newprofile.replace_int_array('secret_list_new', 32, item.child_value('secret_new'))
+            newprofile.replace_int_array('title_list_new', 96, item.child_value('title_new'))
+            newprofile.replace_int('theme_list_new', item.child_value('theme_new'))
+            newprofile.replace_int_array('marker_list_new', 2, item.child_value('marker_new'))
+
+        # Grab macchiato event
+        macchiatodict = newprofile.get_dict('macchiato')
+        macchiato = player.child('macchiato')
+        if macchiato is not None:
+            macchiatodict.replace_int('pack_id', macchiato.child_value('pack_id'))
+            macchiatodict.replace_int('bean_num', macchiato.child_value('bean_num'))
+            macchiatodict.replace_int('daily_milk_num', macchiato.child_value('daily_milk_num'))
+            macchiatodict.replace_bool('is_received_daily_milk', macchiato.child_value('is_received_daily_milk'))
+            macchiatodict.replace_bool('sub_menu_is_completed', macchiato.child_value('sub_menu_is_completed'))
+            macchiatodict.replace_int('today_tune_cnt', macchiato.child_value('today_tune_cnt'))
+            macchiatodict.replace_int_array('daily_milk_bonus', 9, macchiato.child_value('daily_milk_bonus'))
+            macchiatodict.replace_int('compensation_milk', macchiato.child_value('compensation_milk'))
+            macchiatodict.replace_int('match_cnt', macchiato.child_value('match_cnt'))
+            macchiatodict.replace_int('used_bean', macchiato.child_value('used_bean'))
+            macchiatodict.replace_int('used_milk', macchiato.child_value('used_milk'))
+            macchiatodict.replace_int('daily_play_burst', macchiato.child_value('daily_play_burst'))
+        newprofile.replace_dict('macchiato', macchiatodict)
+
+        # Get timestamps for played songs
+        timestamps: Dict[int, int] = {}
+        history = player.child('history')
+        if history is not None:
+            for tune in history.children:
+                if tune.name != 'tune':
+                    continue
+                entry = int(tune.attribute('log_id'))
+                ts = int(tune.child_value('timestamp') / 1000)
+                timestamps[entry] = ts
+
+        # Grab scores and save those
+        result = data.child('result')
+        if result is not None:
+            for tune in result.children:
+                if tune.name != 'tune':
+                    continue
+                result = tune.child('player')
+
+                last.replace_int('marker', tune.child_value('marker'))
+                last.replace_int('title', tune.child_value('title'))
+                last.replace_int('parts', tune.child_value('parts'))
+                last.replace_int('theme', tune.child_value('theme'))
+                last.replace_int('sort', tune.child_value('sort'))
+                last.replace_int('category', tune.child_value('category'))
+                last.replace_int('rank_sort', tune.child_value('rank_sort'))
+                last.replace_int('combo_disp', tune.child_value('combo_disp'))
+
+                songid = tune.child_value('music')
+                entry = int(tune.attribute('id'))
+                timestamp = timestamps.get(entry, Time.now())
+                chart = int(result.child('score').attribute('seq'))
+                points = result.child_value('score')
+                flags = int(result.child('score').attribute('clear'))
+                combo = int(result.child('score').attribute('combo'))
+                ghost = result.child_value('mbar')
+
+                # Miscelaneous last data for echoing to profile get
+                last.replace_int('music_id', songid)
+                last.replace_int('seq_id', chart)
+
+                mapping = {
+                    self.GAME_FLAG_BIT_CLEARED: self.PLAY_MEDAL_CLEARED,
+                    self.GAME_FLAG_BIT_FULL_COMBO: self.PLAY_MEDAL_FULL_COMBO,
+                    self.GAME_FLAG_BIT_EXCELLENT: self.PLAY_MEDAL_EXCELLENT,
+                    self.GAME_FLAG_BIT_NEARLY_FULL_COMBO: self.PLAY_MEDAL_NEARLY_FULL_COMBO,
+                    self.GAME_FLAG_BIT_NEARLY_EXCELLENT: self.PLAY_MEDAL_NEARLY_EXCELLENT,
+                }
+
+                # Figure out the highest medal based on bits passed in
+                medal = self.PLAY_MEDAL_FAILED
+                for bit in mapping:
+                    if flags & bit > 0:
+                        medal = max(medal, mapping[bit])
+
+                self.update_score(userid, timestamp, songid, chart, points, medal, combo, ghost)
+
+        # Grab the course results as well
+        course = data.child('course')
+        if course is not None:
+            courseid = course.child_value('course_id')
+            rating = {
+                self.GAME_COURSE_RATING_FAILED: self.COURSE_RATING_FAILED,
+                self.GAME_COURSE_RATING_BRONZE: self.COURSE_RATING_BRONZE,
+                self.GAME_COURSE_RATING_SILVER: self.COURSE_RATING_SILVER,
+                self.GAME_COURSE_RATING_GOLD: self.COURSE_RATING_GOLD,
+            }[course.child_value('rating')]
+            scores = [0] * 5
+            for music in course.children:
+                if music.name != 'music':
+                    continue
+                index = int(music.attribute('index'))
+                scores[index] = music.child_value('score')
+
+            # Save course itself
+            self.save_course(userid, courseid, rating, scores)
+
+            # Save the last course ID
+            last.replace_int('last_course_id', courseid)
+
+        # Save back last information gleaned from results
+        newprofile.replace_dict('last', last)
+
+        # Keep track of play statistics
+        self.update_play_statistics(userid)
+
+        return newprofile
+
+    def format_scores(self, userid: UserID, profile: ValidatedDict, scores: List[Score]) -> Node:
+        scores = self.data.remote.music.get_scores(self.game, self.version, userid)
+
+        root = Node.void('gametop')
+        datanode = Node.void('data')
+        root.add_child(datanode)
+        player = Node.void('player')
+        datanode.add_child(player)
+        playdata = Node.void('playdata')
+        player.add_child(playdata)
+        playdata.set_attribute('count', str(len(scores)))
+
+        music = ValidatedDict()
+        for score in scores:
+            data = music.get_dict(str(score.id))
+            play_cnt = data.get_int_array('play_cnt', 3)
+            clear_cnt = data.get_int_array('clear_cnt', 3)
+            clear_flags = data.get_int_array('clear_flags', 3)
+            fc_cnt = data.get_int_array('fc_cnt', 3)
+            ex_cnt = data.get_int_array('ex_cnt', 3)
+            points = data.get_int_array('points', 3)
+
+            # Replace data for this chart type
+            play_cnt[score.chart] = score.plays
+            clear_cnt[score.chart] = score.data.get_int('clear_count')
+            fc_cnt[score.chart] = score.data.get_int('full_combo_count')
+            ex_cnt[score.chart] = score.data.get_int('excellent_count')
+            points[score.chart] = score.points
+
+            # Format the clear flags
+            clear_flags[score.chart] = self.GAME_FLAG_BIT_PLAYED
+            if score.data.get_int('clear_count') > 0:
+                clear_flags[score.chart] |= self.GAME_FLAG_BIT_CLEARED
+            if score.data.get_int('full_combo_count') > 0:
+                clear_flags[score.chart] |= self.GAME_FLAG_BIT_FULL_COMBO
+            if score.data.get_int('excellent_count') > 0:
+                clear_flags[score.chart] |= self.GAME_FLAG_BIT_EXCELLENT
+
+            # Save chart data back
+            data.replace_int_array('play_cnt', 3, play_cnt)
+            data.replace_int_array('clear_cnt', 3, clear_cnt)
+            data.replace_int_array('clear_flags', 3, clear_flags)
+            data.replace_int_array('fc_cnt', 3, fc_cnt)
+            data.replace_int_array('ex_cnt', 3, ex_cnt)
+            data.replace_int_array('points', 3, points)
+
+            # Update the ghost (untyped)
+            ghost = data.get('ghost', [None, None, None])
+            ghost[score.chart] = score.data.get('ghost')
+            data['ghost'] = ghost
+
+            # Save it back
+            music.replace_dict(str(score.id), data)
+
+        for scoreid in music:
+            scoredata = music[scoreid]
+            musicdata = Node.void('musicdata')
+            playdata.add_child(musicdata)
+
+            musicdata.set_attribute('music_id', scoreid)
+            musicdata.add_child(Node.s32_array('play_cnt', scoredata.get_int_array('play_cnt', 3)))
+            musicdata.add_child(Node.s32_array('clear_cnt', scoredata.get_int_array('clear_cnt', 3)))
+            musicdata.add_child(Node.s32_array('fc_cnt', scoredata.get_int_array('fc_cnt', 3)))
+            musicdata.add_child(Node.s32_array('ex_cnt', scoredata.get_int_array('ex_cnt', 3)))
+            musicdata.add_child(Node.s32_array('score', scoredata.get_int_array('points', 3)))
+            musicdata.add_child(Node.s8_array('clear', scoredata.get_int_array('clear_flags', 3)))
+
+            ghosts = scoredata.get('ghost', [None, None, None])
+            for i in range(len(ghosts)):
+                ghost = ghosts[i]
+                if ghost is None:
+                    continue
+
+                bar = Node.u8_array('bar', ghost)
+                musicdata.add_child(bar)
+                bar.set_attribute('seq', str(i))
+
+        return root

bemani/backend/jubeat/stubs.py

@@ -0,0 +1,65 @@
+# vim: set fileencoding=utf-8
+from typing import Optional
+
+from bemani.backend.jubeat.base import JubeatBase
+from bemani.common import VersionConstants
+
+
+class Jubeat(JubeatBase):
+
+    name = 'Jubeat'
+    version = VersionConstants.JUBEAT
+
+
+class JubeatRipples(JubeatBase):
+
+    name = 'Jubeat Ripples'
+    version = VersionConstants.JUBEAT_RIPPLES
+
+    def previous_version(self) -> Optional[JubeatBase]:
+        return Jubeat(self.data, self.config, self.model)
+
+
+class JubeatRipplesAppend(JubeatBase):
+
+    name = 'Jubeat Ripples Append'
+    version = VersionConstants.JUBEAT_RIPPLES_APPEND
+
+    def previous_version(self) -> Optional[JubeatBase]:
+        return JubeatRipples(self.data, self.config, self.model)
+
+
+class JubeatKnit(JubeatBase):
+
+    name = 'Jubeat Knit'
+    version = VersionConstants.JUBEAT_KNIT
+
+    def previous_version(self) -> Optional[JubeatBase]:
+        return JubeatRipplesAppend(self.data, self.config, self.model)
+
+
+class JubeatKnitAppend(JubeatBase):
+
+    name = 'Jubeat Knit Append'
+    version = VersionConstants.JUBEAT_KNIT_APPEND
+
+    def previous_version(self) -> Optional[JubeatBase]:
+        return JubeatKnit(self.data, self.config, self.model)
+
+
+class JubeatCopious(JubeatBase):
+
+    name = 'Jubeat Copious'
+    version = VersionConstants.JUBEAT_COPIOUS
+
+    def previous_version(self) -> Optional[JubeatBase]:
+        return JubeatKnitAppend(self.data, self.config, self.model)
+
+
+class JubeatCopiousAppend(JubeatBase):
+
+    name = 'Jubeat Copious Append'
+    version = VersionConstants.JUBEAT_COPIOUS_APPEND
+
+    def previous_version(self) -> Optional[JubeatBase]:
+        return JubeatCopious(self.data, self.config, self.model)

bemani/backend/museca/__init__.py

@@ -0,0 +1,2 @@
+from bemani.backend.museca.factory import MusecaFactory
+from bemani.backend.museca.base import MusecaBase

bemani/backend/museca/base.py

@@ -0,0 +1,286 @@
+# vim: set fileencoding=utf-8
+from typing import Dict, Optional
+
+from bemani.backend.base import Base
+from bemani.backend.core import CoreHandler, CardManagerHandler, PASELIHandler
+from bemani.common import ValidatedDict, GameConstants, DBConstants, Parallel
+from bemani.data import UserID
+from bemani.protocol import Node
+
+
+class MusecaBase(CoreHandler, CardManagerHandler, PASELIHandler, Base):
+    """
+    Base game class for all Museca version that we support.
+    """
+
+    game = GameConstants.MUSECA
+
+    CHART_TYPE_GREEN = 0
+    CHART_TYPE_ORANGE = 1
+    CHART_TYPE_RED = 2
+
+    GRADE_DEATH = DBConstants.MUSECA_GRADE_DEATH
+    GRADE_POOR = DBConstants.MUSECA_GRADE_POOR
+    GRADE_MEDIOCRE = DBConstants.MUSECA_GRADE_MEDIOCRE
+    GRADE_GOOD = DBConstants.MUSECA_GRADE_GOOD
+    GRADE_GREAT = DBConstants.MUSECA_GRADE_GREAT
+    GRADE_EXCELLENT = DBConstants.MUSECA_GRADE_EXCELLENT
+    GRADE_SUPERB = DBConstants.MUSECA_GRADE_SUPERB
+    GRADE_MASTERPIECE = DBConstants.MUSECA_GRADE_MASTERPIECE
+    GRADE_PERFECT = DBConstants.MUSECA_GRADE_PERFECT
+
+    CLEAR_TYPE_FAILED = DBConstants.MUSECA_CLEAR_TYPE_FAILED
+    CLEAR_TYPE_CLEARED = DBConstants.MUSECA_CLEAR_TYPE_CLEARED
+    CLEAR_TYPE_FULL_COMBO = DBConstants.MUSECA_CLEAR_TYPE_FULL_COMBO
+
+    def previous_version(self) -> Optional['MusecaBase']:
+        """
+        Returns the previous version of the game, based on this game. Should
+        be overridden.
+        """
+        return None
+
+    def game_to_db_clear_type(self, clear_type: int) -> int:
+        # Given a game clear type, return the canonical database identifier.
+        raise Exception('Implement in subclass!')
+
+    def db_to_game_clear_type(self, clear_type: int) -> int:
+        # Given a database clear type, return the game's identifier.
+        raise Exception('Implement in subclass!')
+
+    def game_to_db_grade(self, grade: int) -> int:
+        # Given a game grade, return the canonical database identifier.
+        raise Exception('Implement in subclass!')
+
+    def db_to_game_grade(self, grade: int) -> int:
+        # Given a database grade, return the game's identifier.
+        raise Exception('Implement in subclass!')
+
+    def get_profile_by_refid(self, refid: Optional[str]) -> Optional[Node]:
+        """
+        Given a RefID, return a formatted profile node. Basically every game
+        needs a profile lookup, even if it handles where that happens in
+        a different request. This is provided for code deduplication.
+        """
+        if refid is None:
+            return None
+
+        # First try to load the actual profile
+        userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        profile = self.get_profile(userid)
+        if profile is None:
+            return None
+
+        # Now, return it
+        return self.format_profile(userid, profile)
+
+    def new_profile_by_refid(self, refid: Optional[str], name: Optional[str], locid: Optional[int]) -> Node:
+        """
+        Given a RefID and an optional name, create a profile and then return
+        a formatted profile node. Similar rationale to get_profile_by_refid.
+        """
+        if refid is None:
+            return None
+
+        if name is None:
+            name = 'NONAME'
+
+        # First, create and save the default profile
+        userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        defaultprofile = ValidatedDict({
+            'name': name,
+            'loc': locid,
+        })
+        self.put_profile(userid, defaultprofile)
+
+        # Now, reload and format the profile, looking up the has old version flag
+        profile = self.get_profile(userid)
+        return self.format_profile(userid, profile)
+
+    def format_profile(self, userid: UserID, profile: ValidatedDict) -> Node:
+        """
+        Base handler for a profile. Given a userid and a profile dictionary,
+        return a Node representing a profile. Should be overridden.
+        """
+        return Node.void('game')
+
+    def unformat_profile(self, userid: UserID, request: Node, oldprofile: ValidatedDict) -> ValidatedDict:
+        """
+        Base handler for profile parsing. Given a request and an old profile,
+        return a new profile that's been updated with the contents of the request.
+        Should be overridden.
+        """
+        return oldprofile
+
+    def get_clear_rates(self) -> Dict[int, Dict[int, Dict[str, int]]]:
+        """
+        Returns a dictionary similar to the following:
+
+        {
+            musicid: {
+                chart: {
+                    total: total plays,
+                    clears: total clears,
+                },
+            },
+        }
+        """
+        all_attempts, remote_attempts = Parallel.execute([
+            lambda: self.data.local.music.get_all_attempts(
+                game=self.game,
+                version=self.version,
+            ),
+            lambda: self.data.remote.music.get_clear_rates(
+                game=self.game,
+                version=self.version,
+            )
+        ])
+        attempts: Dict[int, Dict[int, Dict[str, int]]] = {}
+        for (_, attempt) in all_attempts:
+            # Terrible temporary structure is terrible.
+            if attempt.id not in attempts:
+                attempts[attempt.id] = {}
+            if attempt.chart not in attempts[attempt.id]:
+                attempts[attempt.id][attempt.chart] = {
+                    'total': 0,
+                    'clears': 0,
+                }
+
+            # We saw an attempt, keep the total attempts in sync.
+            attempts[attempt.id][attempt.chart]['total'] = attempts[attempt.id][attempt.chart]['total'] + 1
+
+            if attempt.data.get_int('clear_type', self.CLEAR_TYPE_FAILED) != self.CLEAR_TYPE_FAILED:
+                # This attempt was a failure, so don't count it against clears of full combos
+                continue
+
+            # It was at least a clear
+            attempts[attempt.id][attempt.chart]['clears'] = attempts[attempt.id][attempt.chart]['clears'] + 1
+
+        # Merge in remote attempts
+        for songid in remote_attempts:
+            if songid not in attempts:
+                attempts[songid] = {}
+
+            for songchart in remote_attempts[songid]:
+                if songchart not in attempts[songid]:
+                    attempts[songid][songchart] = {
+                        'total': 0,
+                        'clears': 0,
+                    }
+
+                attempts[songid][songchart]['total'] += remote_attempts[songid][songchart]['plays']
+                attempts[songid][songchart]['clears'] += remote_attempts[songid][songchart]['clears']
+
+        return attempts
+
+    def update_score(
+        self,
+        userid: Optional[UserID],
+        songid: int,
+        chart: int,
+        points: int,
+        clear_type: int,
+        grade: int,
+        combo: int,
+        stats: Optional[Dict[str, int]]=None,
+    ) -> None:
+        """
+        Given various pieces of a score, update the user's high score and score
+        history in a controlled manner, so all games in SDVX series can expect
+        the same attributes in a score.
+        """
+        # Range check clear type
+        if clear_type not in [
+            self.CLEAR_TYPE_FAILED,
+            self.CLEAR_TYPE_CLEARED,
+            self.CLEAR_TYPE_FULL_COMBO,
+        ]:
+            raise Exception("Invalid clear type value {}".format(clear_type))
+
+        # Range check grade
+        if grade not in [
+            self.GRADE_DEATH,
+            self.GRADE_POOR,
+            self.GRADE_MEDIOCRE,
+            self.GRADE_GOOD,
+            self.GRADE_GREAT,
+            self.GRADE_EXCELLENT,
+            self.GRADE_SUPERB,
+            self.GRADE_MASTERPIECE,
+            self.GRADE_PERFECT,
+        ]:
+            raise Exception("Invalid grade value {}".format(grade))
+
+        if userid is not None:
+            oldscore = self.data.local.music.get_score(
+                self.game,
+                self.version,
+                userid,
+                songid,
+                chart,
+            )
+        else:
+            oldscore = None
+
+        # Score history is verbatum, instead of highest score
+        history = ValidatedDict({})
+        oldpoints = points
+
+        if oldscore is None:
+            # If it is a new score, create a new dictionary to add to
+            scoredata = ValidatedDict({})
+            raised = True
+            highscore = True
+        else:
+            # Set the score to any new record achieved
+            raised = points > oldscore.points
+            highscore = points >= oldscore.points
+            points = max(oldscore.points, points)
+            scoredata = oldscore.data
+
+        # Replace grade and clear type
+        scoredata.replace_int('clear_type', max(scoredata.get_int('clear_type'), clear_type))
+        history.replace_int('clear_type', clear_type)
+        scoredata.replace_int('grade', max(scoredata.get_int('grade'), grade))
+        history.replace_int('grade', grade)
+
+        # If we have a combo, replace it
+        scoredata.replace_int('combo', max(scoredata.get_int('combo'), combo))
+        history.replace_int('combo', combo)
+
+        # If we have play stats, replace it
+        if stats is not None:
+            if raised:
+                # We have stats, and there's a new high score, update the stats
+                scoredata.replace_dict('stats', stats)
+            history.replace_dict('stats', stats)
+
+        # Look up where this score was earned
+        lid = self.get_machine_id()
+
+        if userid is not None:
+            # Write the new score back
+            self.data.local.music.put_score(
+                self.game,
+                self.version,
+                userid,
+                songid,
+                chart,
+                lid,
+                points,
+                scoredata,
+                highscore,
+            )
+
+        # Save the history of this score too
+        self.data.local.music.put_attempt(
+            self.game,
+            self.version,
+            userid,
+            songid,
+            chart,
+            lid,
+            oldpoints,
+            history,
+            raised,
+        )

bemani/backend/museca/common.py

@@ -0,0 +1,203 @@
+
+from bemani.backend.museca.base import MusecaBase
+from bemani.common import ID
+from bemani.protocol import Node
+
+
+class MusecaGameShopHandler(MusecaBase):
+
+    def handle_game_3_shop_request(self, request: Node) -> Node:
+        self.update_machine_name(request.child_value('shopname'))
+
+        # Respond with number of milliseconds until next request
+        game = Node.void('game_3')
+        game.add_child(Node.u32('nxt_time', 1000 * 5 * 60))
+        return game
+
+
+class MusecaGameHiscoreHandler(MusecaBase):
+
+    def handle_game_3_hiscore_request(self, request: Node) -> Node:
+        # Grab location for local scores
+        locid = ID.parse_machine_id(request.child_value('locid'))
+
+        # Start the response packet
+        game = Node.void('game_3')
+
+        # First, grab hit chart
+        playcounts = self.data.local.music.get_hit_chart(self.game, self.version, 1024)
+
+        hitchart = Node.void('hitchart')
+        game.add_child(hitchart)
+        for (songid, count) in playcounts:
+            info = Node.void('info')
+            hitchart.add_child(info)
+            info.add_child(Node.u32('id', songid))
+            info.add_child(Node.u32('cnt', count))
+
+        # Now, grab user records
+        records = self.data.remote.music.get_all_records(self.game, self.version)
+        users = {
+            uid: prof for (uid, prof) in
+            self.get_any_profiles([r[0] for r in records])
+        }
+
+        hiscore_allover = Node.void('hiscore_allover')
+        game.add_child(hiscore_allover)
+
+        # Output records
+        for (userid, score) in records:
+            info = Node.void('info')
+
+            if userid not in users:
+                raise Exception('Logic error, could not find profile for user!')
+            profile = users[userid]
+
+            info.add_child(Node.u32('id', score.id))
+            info.add_child(Node.u32('type', score.chart))
+            info.add_child(Node.string('name', profile.get_str('name')))
+            info.add_child(Node.string('seq', ID.format_extid(profile.get_int('extid'))))
+            info.add_child(Node.u32('score', score.points))
+
+            # Add to global scores
+            hiscore_allover.add_child(info)
+
+        # Now, grab local records
+        area_users = [
+            uid for (uid, prof) in self.data.local.user.get_all_profiles(self.game, self.version)
+            if prof.get_int('loc', -1) == locid
+        ]
+        records = self.data.local.music.get_all_records(self.game, self.version, userlist=area_users)
+        missing_players = [
+            uid for (uid, _) in records
+            if uid not in users
+        ]
+        for (uid, prof) in self.get_any_profiles(missing_players):
+            users[uid] = prof
+
+        hiscore_location = Node.void('hiscore_location')
+        game.add_child(hiscore_location)
+
+        # Output records
+        for (userid, score) in records:
+            info = Node.void('info')
+
+            if userid not in users:
+                raise Exception('Logic error, could not find profile for user!')
+            profile = users[userid]
+
+            info.add_child(Node.u32('id', score.id))
+            info.add_child(Node.u32('type', score.chart))
+            info.add_child(Node.string('name', profile.get_str('name')))
+            info.add_child(Node.string('seq', ID.format_extid(profile.get_int('extid'))))
+            info.add_child(Node.u32('score', score.points))
+
+            # Add to global scores
+            hiscore_location.add_child(info)
+
+        # Now, grab clear rates
+        clear_rate = Node.void('clear_rate')
+        game.add_child(clear_rate)
+
+        clears = self.get_clear_rates()
+        for songid in clears:
+            for chart in clears[songid]:
+                if clears[songid][chart]['total'] > 0:
+                    rate = float(clears[songid][chart]['clears']) / float(clears[songid][chart]['total'])
+                    dnode = Node.void('d')
+                    clear_rate.add_child(dnode)
+                    dnode.add_child(Node.u32('id', songid))
+                    dnode.add_child(Node.u32('type', chart))
+                    dnode.add_child(Node.s16('cr', int(rate * 10000)))
+
+        return game
+
+
+class MusecaGameFrozenHandler(MusecaBase):
+
+    def handle_game_3_frozen_request(self, request: Node) -> Node:
+        game = Node.void('game_3')
+        game.add_child(Node.u8('result', 0))
+        return game
+
+
+class MusecaGameNewHandler(MusecaBase):
+
+    def handle_game_3_new_request(self, request: Node) -> Node:
+        refid = request.child_value('refid')
+        name = request.child_value('name')
+        loc = ID.parse_machine_id(request.child_value('locid'))
+        self.new_profile_by_refid(refid, name, loc)
+
+        root = Node.void('game_3')
+        return root
+
+
+class MusecaGameSaveMusicHandler(MusecaBase):
+
+    def handle_game_3_save_m_request(self, request: Node) -> Node:
+        refid = request.child_value('refid')
+
+        if refid is not None:
+            userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        else:
+            userid = None
+
+        # Doesn't matter if userid is None here, that's an anonymous score
+        musicid = request.child_value('music_id')
+        chart = request.child_value('music_type')
+        points = request.child_value('score')
+        combo = request.child_value('max_chain')
+        clear_type = self.game_to_db_clear_type(request.child_value('clear_type'))
+        grade = self.game_to_db_grade(request.child_value('score_grade'))
+        stats = {
+            'btn_rate': request.child_value('btn_rate'),
+            'long_rate': request.child_value('long_rate'),
+            'vol_rate': request.child_value('vol_rate'),
+            'critical': request.child_value('critical'),
+            'near': request.child_value('near'),
+            'error': request.child_value('error'),
+        }
+
+        # Save the score
+        self.update_score(
+            userid,
+            musicid,
+            chart,
+            points,
+            clear_type,
+            grade,
+            combo,
+            stats,
+        )
+
+        # Return a blank response
+        return Node.void('game_3')
+
+
+class MusecaGamePlayEndHandler(MusecaBase):
+
+    def handle_game_3_play_e_request(self, request: Node) -> Node:
+        return Node.void('game_3')
+
+
+class MusecaGameSaveHandler(MusecaBase):
+
+    def handle_game_3_save_request(self, request: Node) -> Node:
+        refid = request.child_value('refid')
+
+        if refid is not None:
+            userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        else:
+            userid = None
+
+        if userid is not None:
+            oldprofile = self.get_profile(userid)
+            newprofile = self.unformat_profile(userid, request, oldprofile)
+        else:
+            newprofile = None
+
+        if userid is not None and newprofile is not None:
+            self.put_profile(userid, newprofile)
+
+        return Node.void('game_3')

bemani/backend/museca/factory.py

@@ -0,0 +1,40 @@
+from typing import Any, Dict, List, Optional, Type
+
+from bemani.backend.base import Base, Factory
+from bemani.backend.museca.museca1 import Museca1
+from bemani.backend.museca.museca1plus import Museca1Plus
+from bemani.common import Model, VersionConstants
+from bemani.data import Data
+
+
+class MusecaFactory(Factory):
+
+    MANAGED_CLASSES: List[Type[Base]] = [
+        Museca1,
+        Museca1Plus,
+    ]
+
+    @classmethod
+    def register_all(cls) -> None:
+        for game in ['PIX']:
+            Base.register(game, MusecaFactory)
+
+    @classmethod
+    def create(cls, data: Data, config: Dict[str, Any], model: Model, parentmodel: Optional[Model]=None) -> Optional[Base]:
+
+        def version_from_date(date: int) -> Optional[int]:
+            if date <= 2016072600:
+                return VersionConstants.MUSECA
+            if date > 2016072600:
+                return VersionConstants.MUSECA_1_PLUS
+            return None
+
+        if model.game == 'PIX':
+            version = version_from_date(model.version)
+            if version == VersionConstants.MUSECA:
+                return Museca1(data, config, model)
+            if version == VersionConstants.MUSECA_1_PLUS:
+                return Museca1Plus(data, config, model)
+
+        # Unknown game version
+        return None

bemani/backend/museca/museca1.py

@@ -0,0 +1,364 @@
+import copy
+from typing import Any, Dict
+
+from bemani.backend.ess import EventLogHandler
+from bemani.backend.museca.base import MusecaBase
+from bemani.backend.museca.common import (
+    MusecaGameFrozenHandler,
+    MusecaGameHiscoreHandler,
+    MusecaGameNewHandler,
+    MusecaGamePlayEndHandler,
+    MusecaGameSaveHandler,
+    MusecaGameSaveMusicHandler,
+    MusecaGameShopHandler,
+)
+from bemani.common import Time, VersionConstants, ValidatedDict, ID
+from bemani.data import UserID
+from bemani.protocol import Node
+
+
+class Museca1(
+    EventLogHandler,
+    MusecaGameFrozenHandler,
+    MusecaGameHiscoreHandler,
+    MusecaGameNewHandler,
+    MusecaGamePlayEndHandler,
+    MusecaGameSaveHandler,
+    MusecaGameSaveMusicHandler,
+    MusecaGameShopHandler,
+    MusecaBase,
+):
+
+    name = "MÚSECA"
+    version = VersionConstants.MUSECA
+
+    GAME_LIMITED_LOCKED = 1
+    GAME_LIMITED_UNLOCKABLE = 2
+    GAME_LIMITED_UNLOCKED = 3
+
+    GAME_CATALOG_TYPE_SONG = 0
+    GAME_CATALOG_TYPE_GRAFICA = 15
+    GAME_CATALOG_TYPE_MISSION = 16
+
+    GAME_GRADE_DEATH = 0
+    GAME_GRADE_POOR = 1
+    GAME_GRADE_MEDIOCRE = 2
+    GAME_GRADE_GOOD = 3
+    GAME_GRADE_GREAT = 4
+    GAME_GRADE_EXCELLENT = 5
+    GAME_GRADE_SUPERB = 6
+    GAME_GRADE_MASTERPIECE = 7
+
+    GAME_CLEAR_TYPE_FAILED = 1
+    GAME_CLEAR_TYPE_CLEARED = 2
+    GAME_CLEAR_TYPE_FULL_COMBO = 4
+
+    @classmethod
+    def get_settings(cls) -> Dict[str, Any]:
+        """
+        Return all of our front-end modifiably settings.
+        """
+        return {
+            'bools': [
+                {
+                    'name': 'Force Song Unlock',
+                    'tip': 'Force unlock all songs.',
+                    'category': 'game_config',
+                    'setting': 'force_unlock_songs',
+                },
+            ],
+        }
+
+    def game_to_db_clear_type(self, clear_type: int) -> int:
+        return {
+            self.GAME_CLEAR_TYPE_FAILED: self.CLEAR_TYPE_FAILED,
+            self.GAME_CLEAR_TYPE_CLEARED: self.CLEAR_TYPE_CLEARED,
+            self.GAME_CLEAR_TYPE_FULL_COMBO: self.CLEAR_TYPE_FULL_COMBO,
+        }[clear_type]
+
+    def db_to_game_clear_type(self, clear_type: int) -> int:
+        return {
+            self.CLEAR_TYPE_FAILED: self.GAME_CLEAR_TYPE_FAILED,
+            self.CLEAR_TYPE_CLEARED: self.GAME_CLEAR_TYPE_CLEARED,
+            self.CLEAR_TYPE_FULL_COMBO: self.GAME_CLEAR_TYPE_FULL_COMBO,
+        }[clear_type]
+
+    def game_to_db_grade(self, grade: int) -> int:
+        return {
+            self.GAME_GRADE_DEATH: self.GRADE_DEATH,
+            self.GAME_GRADE_POOR: self.GRADE_POOR,
+            self.GAME_GRADE_MEDIOCRE: self.GRADE_MEDIOCRE,
+            self.GAME_GRADE_GOOD: self.GRADE_GOOD,
+            self.GAME_GRADE_GREAT: self.GRADE_GREAT,
+            self.GAME_GRADE_EXCELLENT: self.GRADE_EXCELLENT,
+            self.GAME_GRADE_SUPERB: self.GRADE_SUPERB,
+            self.GAME_GRADE_MASTERPIECE: self.GRADE_MASTERPIECE,
+        }[grade]
+
+    def db_to_game_grade(self, grade: int) -> int:
+        return {
+            self.GRADE_DEATH: self.GAME_GRADE_DEATH,
+            self.GRADE_POOR: self.GAME_GRADE_POOR,
+            self.GRADE_MEDIOCRE: self.GAME_GRADE_MEDIOCRE,
+            self.GRADE_GOOD: self.GAME_GRADE_GOOD,
+            self.GRADE_GREAT: self.GAME_GRADE_GREAT,
+            self.GRADE_EXCELLENT: self.GAME_GRADE_EXCELLENT,
+            self.GRADE_SUPERB: self.GAME_GRADE_SUPERB,
+            self.GRADE_MASTERPIECE: self.GAME_GRADE_MASTERPIECE,
+            self.GRADE_PERFECT: self.GAME_GRADE_MASTERPIECE,
+        }[grade]
+
+    def handle_game_3_common_request(self, request: Node) -> Node:
+        game = Node.void('game_3')
+        limited = Node.void('music_limited')
+        game.add_child(limited)
+
+        # Song unlock config
+        game_config = self.get_game_config()
+        if game_config.get_bool('force_unlock_songs'):
+            ids = set()
+            songs = self.data.local.music.get_all_songs(self.game, self.version)
+            for song in songs:
+                if song.data.get_int('limited') in (self.GAME_LIMITED_LOCKED, self.GAME_LIMITED_UNLOCKABLE):
+                    ids.add((song.id, song.chart))
+
+            for (songid, chart) in ids:
+                info = Node.void('info')
+                limited.add_child(info)
+                info.add_child(Node.s32('music_id', songid))
+                info.add_child(Node.u8('music_type', chart))
+                info.add_child(Node.u8('limited', self.GAME_LIMITED_UNLOCKED))
+
+        # Event config
+        event = Node.void('event')
+        game.add_child(event)
+
+        def enable_event(eid: int) -> None:
+            evt = Node.void('info')
+            event.add_child(evt)
+            evt.add_child(Node.u32('event_id', eid))
+
+        # Allow PASELI light start
+        enable_event(83)
+
+        # If you want song unlock news to show up, enable one of the following:
+        # 94 - 5/25/2016 unlocks
+        # 95 - 4/27/2016 second unlocks
+        # 89 - 4/27/2016 unlocks
+        # 87 - 4/13/2016 unlocks
+        # 82 - 3/23/2016 second unlocks
+        # 80 - 3/23/2016 unlocks
+        # 76 - 12/22/2016 unlocks
+
+        return game
+
+    def handle_game_3_exception_request(self, request: Node) -> Node:
+        return Node.void('game_3')
+
+    def handle_game_3_load_request(self, request: Node) -> Node:
+        refid = request.child_value('refid')
+        root = self.get_profile_by_refid(refid)
+        if root is not None:
+            return root
+
+        # No data succession, there's nothing older than this!
+        root = Node.void('game_3')
+        root.add_child(Node.u8('result', 1))
+        return root
+
+    def handle_game_3_load_m_request(self, request: Node) -> Node:
+        refid = request.child_value('dataid')
+
+        if refid is not None:
+            userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        else:
+            userid = None
+
+        if userid is not None:
+            scores = self.data.remote.music.get_scores(self.game, self.version, userid)
+        else:
+            scores = []
+
+        # Output to the game
+        game = Node.void('game_3')
+        new = Node.void('new')
+        game.add_child(new)
+
+        for score in scores:
+            music = Node.void('music')
+            new.add_child(music)
+            music.add_child(Node.u32('music_id', score.id))
+            music.add_child(Node.u32('music_type', score.chart))
+            music.add_child(Node.u32('score', score.points))
+            music.add_child(Node.u32('cnt', score.plays))
+            music.add_child(Node.u32('clear_type', self.db_to_game_clear_type(score.data.get_int('clear_type'))))
+            music.add_child(Node.u32('score_grade', self.db_to_game_grade(score.data.get_int('grade'))))
+            stats = score.data.get_dict('stats')
+            music.add_child(Node.u32('btn_rate', stats.get_int('btn_rate')))
+            music.add_child(Node.u32('long_rate', stats.get_int('long_rate')))
+            music.add_child(Node.u32('vol_rate', stats.get_int('vol_rate')))
+
+        return game
+
+    def format_profile(self, userid: UserID, profile: ValidatedDict) -> Node:
+        game = Node.void('game_3')
+
+        # Generic profile stuff
+        game.add_child(Node.string('name', profile.get_str('name')))
+        game.add_child(Node.string('code', ID.format_extid(profile.get_int('extid'))))
+        game.add_child(Node.u32('gamecoin_packet', profile.get_int('packet')))
+        game.add_child(Node.u32('gamecoin_block', profile.get_int('block')))
+        game.add_child(Node.s16('skill_name_id', profile.get_int('skill_name_id', -1)))
+        game.add_child(Node.s32_array('hidden_param', profile.get_int_array('hidden_param', 20)))
+        game.add_child(Node.u32('blaster_energy', profile.get_int('blaster_energy')))
+        game.add_child(Node.u32('blaster_count', profile.get_int('blaster_count')))
+
+        # Play statistics
+        statistics = self.get_play_statistics(userid)
+        last_play_date = statistics.get_int_array('last_play_date', 3)
+        today_play_date = Time.todays_date()
+        if (
+            last_play_date[0] == today_play_date[0] and
+            last_play_date[1] == today_play_date[1] and
+            last_play_date[2] == today_play_date[2]
+        ):
+            today_count = statistics.get_int('today_plays', 0)
+        else:
+            today_count = 0
+        game.add_child(Node.u32('play_count', statistics.get_int('total_plays', 0)))
+        game.add_child(Node.u32('daily_count', today_count))
+        game.add_child(Node.u32('play_chain', statistics.get_int('consecutive_days', 0)))
+
+        # Last played stuff
+        if 'last' in profile:
+            lastdict = profile.get_dict('last')
+            last = Node.void('last')
+            game.add_child(last)
+            last.add_child(Node.s32('music_id', lastdict.get_int('music_id', -1)))
+            last.add_child(Node.u8('music_type', lastdict.get_int('music_type')))
+            last.add_child(Node.u8('sort_type', lastdict.get_int('sort_type')))
+            last.add_child(Node.u8('narrow_down', lastdict.get_int('narrow_down')))
+            last.add_child(Node.u8('headphone', lastdict.get_int('headphone')))
+            last.add_child(Node.u16('appeal_id', lastdict.get_int('appeal_id', 1001)))
+            last.add_child(Node.u16('comment_id', lastdict.get_int('comment_id')))
+            last.add_child(Node.u8('gauge_option', lastdict.get_int('gauge_option')))
+
+        # Item unlocks
+        itemnode = Node.void('item')
+        game.add_child(itemnode)
+
+        game_config = self.get_game_config()
+        achievements = self.data.local.user.get_achievements(self.game, self.version, userid)
+
+        for item in achievements:
+            if item.type[:5] != 'item_':
+                continue
+            itemtype = int(item.type[5:])
+
+            if game_config.get_bool('force_unlock_songs') and itemtype == self.GAME_CATALOG_TYPE_SONG:
+                # Don't echo unlocked songs, we will add all of them later
+                continue
+
+            info = Node.void('info')
+            itemnode.add_child(info)
+            info.add_child(Node.u8('type', itemtype))
+            info.add_child(Node.u32('id', item.id))
+            info.add_child(Node.u32('param', item.data.get_int('param')))
+            if 'diff_param' in item.data:
+                info.add_child(Node.s32('diff_param', item.data.get_int('diff_param')))
+
+        if game_config.get_bool('force_unlock_songs'):
+            ids: Dict[int, int] = {}
+            songs = self.data.local.music.get_all_songs(self.game, self.version)
+            for song in songs:
+                if song.id not in ids:
+                    ids[song.id] = 0
+
+                if song.data.get_int('difficulty') > 0:
+                    ids[song.id] = ids[song.id] | (1 << song.chart)
+
+            for itemid in ids:
+                if ids[itemid] == 0:
+                    continue
+
+                info = Node.void('info')
+                itemnode.add_child(info)
+                info.add_child(Node.u8('type', self.GAME_CATALOG_TYPE_SONG))
+                info.add_child(Node.u32('id', itemid))
+                info.add_child(Node.u32('param', ids[itemid]))
+
+        return game
+
+    def unformat_profile(self, userid: UserID, request: Node, oldprofile: ValidatedDict) -> ValidatedDict:
+        newprofile = copy.deepcopy(oldprofile)
+
+        # Update blaster energy and in-game currencies
+        earned_gamecoin_packet = request.child_value('earned_gamecoin_packet')
+        if earned_gamecoin_packet is not None:
+            newprofile.replace_int('packet', newprofile.get_int('packet') + earned_gamecoin_packet)
+        earned_gamecoin_block = request.child_value('earned_gamecoin_block')
+        if earned_gamecoin_block is not None:
+            newprofile.replace_int('block', newprofile.get_int('block') + earned_gamecoin_block)
+        earned_blaster_energy = request.child_value('earned_blaster_energy')
+        if earned_blaster_energy is not None:
+            newprofile.replace_int('blaster_energy', newprofile.get_int('blaster_energy') + earned_blaster_energy)
+
+        # Miscelaneous stuff
+        newprofile.replace_int('blaster_count', request.child_value('blaster_count'))
+        newprofile.replace_int('skill_name_id', request.child_value('skill_name_id'))
+        newprofile.replace_int_array('hidden_param', 20, request.child_value('hidden_param'))
+
+        # Update user's unlock status if we aren't force unlocked
+        game_config = self.get_game_config()
+
+        if request.child('item') is not None:
+            for child in request.child('item').children:
+                if child.name != 'info':
+                    continue
+
+                item_id = child.child_value('id')
+                item_type = child.child_value('type')
+                param = child.child_value('param')
+                diff_param = child.child_value('diff_param')
+
+                if game_config.get_bool('force_unlock_songs') and item_type == self.GAME_CATALOG_TYPE_SONG:
+                    # Don't save back songs, because they were force unlocked
+                    continue
+
+                if diff_param is not None:
+                    paramvals = {
+                        'diff_param': diff_param,
+                        'param': param,
+                    }
+                else:
+                    paramvals = {
+                        'param': param,
+                    }
+
+                self.data.local.user.put_achievement(
+                    self.game,
+                    self.version,
+                    userid,
+                    item_id,
+                    'item_{}'.format(item_type),
+                    paramvals,
+                )
+
+        # Grab last information.
+        lastdict = newprofile.get_dict('last')
+        lastdict.replace_int('headphone', request.child_value('headphone'))
+        lastdict.replace_int('appeal_id', request.child_value('appeal_id'))
+        lastdict.replace_int('comment_id', request.child_value('comment_id'))
+        lastdict.replace_int('music_id', request.child_value('music_id'))
+        lastdict.replace_int('music_type', request.child_value('music_type'))
+        lastdict.replace_int('sort_type', request.child_value('sort_type'))
+        lastdict.replace_int('narrow_down', request.child_value('narrow_down'))
+        lastdict.replace_int('gauge_option', request.child_value('gauge_option'))
+
+        # Save back last information gleaned from results
+        newprofile.replace_dict('last', lastdict)
+
+        # Keep track of play statistics
+        self.update_play_statistics(userid)
+
+        return newprofile

bemani/backend/museca/museca1plus.py

@@ -0,0 +1,405 @@
+import copy
+from typing import Optional, Dict, Any
+
+from bemani.backend.ess import EventLogHandler
+from bemani.backend.museca.base import MusecaBase
+from bemani.backend.museca.common import (
+    MusecaGameFrozenHandler,
+    MusecaGameHiscoreHandler,
+    MusecaGameNewHandler,
+    MusecaGamePlayEndHandler,
+    MusecaGameSaveHandler,
+    MusecaGameSaveMusicHandler,
+    MusecaGameShopHandler,
+)
+from bemani.backend.museca.museca1 import Museca1
+from bemani.common import Time, VersionConstants, ValidatedDict, ID
+from bemani.data import UserID
+from bemani.protocol import Node
+
+
+class Museca1Plus(
+    EventLogHandler,
+    MusecaGameFrozenHandler,
+    MusecaGameHiscoreHandler,
+    MusecaGameNewHandler,
+    MusecaGamePlayEndHandler,
+    MusecaGameSaveHandler,
+    MusecaGameSaveMusicHandler,
+    MusecaGameShopHandler,
+    MusecaBase,
+):
+
+    name = "MÚSECA 1+1/2"
+    version = VersionConstants.MUSECA_1_PLUS
+
+    GAME_LIMITED_LOCKED = 1
+    GAME_LIMITED_UNLOCKABLE = 2
+    GAME_LIMITED_UNLOCKED = 3
+
+    GAME_CATALOG_TYPE_SONG = 0
+    GAME_CATALOG_TYPE_GRAFICA = 15
+    GAME_CATALOG_TYPE_MISSION = 16
+
+    GAME_GRADE_DEATH = 0
+    GAME_GRADE_POOR = 1
+    GAME_GRADE_MEDIOCRE = 2
+    GAME_GRADE_GOOD = 3
+    GAME_GRADE_GREAT = 4
+    GAME_GRADE_EXCELLENT = 5
+    GAME_GRADE_SUPERB = 6
+    GAME_GRADE_MASTERPIECE = 7
+    GAME_GRADE_PERFECT = 8
+
+    GAME_CLEAR_TYPE_FAILED = 1
+    GAME_CLEAR_TYPE_CLEARED = 2
+    GAME_CLEAR_TYPE_FULL_COMBO = 4
+
+    def previous_version(self) -> Optional[MusecaBase]:
+        return Museca1(self.data, self.config, self.model)
+
+    @classmethod
+    def get_settings(cls) -> Dict[str, Any]:
+        """
+        Return all of our front-end modifiably settings.
+        """
+        return {
+            'bools': [
+                {
+                    'name': 'Disable Online Matching',
+                    'tip': 'Disable online matching between games.',
+                    'category': 'game_config',
+                    'setting': 'disable_matching',
+                },
+                {
+                    'name': 'Force Song Unlock',
+                    'tip': 'Force unlock all songs.',
+                    'category': 'game_config',
+                    'setting': 'force_unlock_songs',
+                },
+            ],
+        }
+
+    def game_to_db_clear_type(self, clear_type: int) -> int:
+        return {
+            self.GAME_CLEAR_TYPE_FAILED: self.CLEAR_TYPE_FAILED,
+            self.GAME_CLEAR_TYPE_CLEARED: self.CLEAR_TYPE_CLEARED,
+            self.GAME_CLEAR_TYPE_FULL_COMBO: self.CLEAR_TYPE_FULL_COMBO,
+        }[clear_type]
+
+    def db_to_game_clear_type(self, clear_type: int) -> int:
+        return {
+            self.CLEAR_TYPE_FAILED: self.GAME_CLEAR_TYPE_FAILED,
+            self.CLEAR_TYPE_CLEARED: self.GAME_CLEAR_TYPE_CLEARED,
+            self.CLEAR_TYPE_FULL_COMBO: self.GAME_CLEAR_TYPE_FULL_COMBO,
+        }[clear_type]
+
+    def game_to_db_grade(self, grade: int) -> int:
+        return {
+            self.GAME_GRADE_DEATH: self.GRADE_DEATH,
+            self.GAME_GRADE_POOR: self.GRADE_POOR,
+            self.GAME_GRADE_MEDIOCRE: self.GRADE_MEDIOCRE,
+            self.GAME_GRADE_GOOD: self.GRADE_GOOD,
+            self.GAME_GRADE_GREAT: self.GRADE_GREAT,
+            self.GAME_GRADE_EXCELLENT: self.GRADE_EXCELLENT,
+            self.GAME_GRADE_SUPERB: self.GRADE_SUPERB,
+            self.GAME_GRADE_MASTERPIECE: self.GRADE_MASTERPIECE,
+            self.GAME_GRADE_PERFECT: self.GRADE_PERFECT,
+        }[grade]
+
+    def db_to_game_grade(self, grade: int) -> int:
+        return {
+            self.GRADE_DEATH: self.GAME_GRADE_DEATH,
+            self.GRADE_POOR: self.GAME_GRADE_POOR,
+            self.GRADE_MEDIOCRE: self.GAME_GRADE_MEDIOCRE,
+            self.GRADE_GOOD: self.GAME_GRADE_GOOD,
+            self.GRADE_GREAT: self.GAME_GRADE_GREAT,
+            self.GRADE_EXCELLENT: self.GAME_GRADE_EXCELLENT,
+            self.GRADE_SUPERB: self.GAME_GRADE_SUPERB,
+            self.GRADE_MASTERPIECE: self.GAME_GRADE_MASTERPIECE,
+            self.GRADE_PERFECT: self.GAME_GRADE_PERFECT,
+        }[grade]
+
+    def handle_game_3_common_request(self, request: Node) -> Node:
+        game = Node.void('game_3')
+        limited = Node.void('music_limited')
+        game.add_child(limited)
+
+        # Song unlock config
+        game_config = self.get_game_config()
+        if game_config.get_bool('force_unlock_songs'):
+            ids = set()
+            songs = self.data.local.music.get_all_songs(self.game, self.version)
+            for song in songs:
+                if song.data.get_int('limited') in (self.GAME_LIMITED_LOCKED, self.GAME_LIMITED_UNLOCKABLE):
+                    ids.add((song.id, song.chart))
+
+            for (songid, chart) in ids:
+                info = Node.void('info')
+                limited.add_child(info)
+                info.add_child(Node.s32('music_id', songid))
+                info.add_child(Node.u8('music_type', chart))
+                info.add_child(Node.u8('limited', self.GAME_LIMITED_UNLOCKED))
+
+        # Event config
+        event = Node.void('event')
+        game.add_child(event)
+
+        def enable_event(eid: int) -> None:
+            evt = Node.void('info')
+            event.add_child(evt)
+            evt.add_child(Node.u32('event_id', eid))
+
+        if not game_config.get_bool('disable_matching'):
+            enable_event(143)  # Matching enabled
+
+        enable_event(1)  # Extended pedal options
+        enable_event(83)  # Light start
+        enable_event(130)  # Curator rank
+        enable_event(195)  # Fictional curator
+        # Event 194 is continuation mode, but it doesn't seem to work on latest data.
+
+        enable_event(98)  # Mission mode
+        for evtid in [145, 146, 147, 148, 149]:
+            enable_event(evtid)  # Mission stuff
+
+        return game
+
+    def handle_game_3_lounge_request(self, request: Node) -> Node:
+        game = Node.void('game_3')
+        # Refresh interval in seconds.
+        game.add_child(Node.u32('interval', 10))
+        return game
+
+    def handle_game_3_exception_request(self, request: Node) -> Node:
+        return Node.void('game_3')
+
+    def handle_game_3_load_request(self, request: Node) -> Node:
+        refid = request.child_value('refid')
+        root = self.get_profile_by_refid(refid)
+        if root is not None:
+            return root
+
+        # Figure out if this user has an older profile or not
+        userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+
+        if userid is not None:
+            previous_game = self.previous_version()
+        else:
+            previous_game = None
+
+        if previous_game is not None:
+            profile = previous_game.get_profile(userid)
+        else:
+            profile = None
+
+        if profile is not None:
+            # Return the previous formatted profile to the game.
+            return previous_game.format_profile(userid, profile)
+        else:
+            root = Node.void('game_3')
+            root.add_child(Node.u8('result', 1))
+            return root
+
+    def handle_game_3_load_m_request(self, request: Node) -> Node:
+        refid = request.child_value('dataid')
+
+        if refid is not None:
+            userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        else:
+            userid = None
+
+        if userid is not None:
+            scores = self.data.remote.music.get_scores(self.game, self.version, userid)
+        else:
+            scores = []
+
+        # Output to the game
+        game = Node.void('game_3')
+        new = Node.void('new')
+        game.add_child(new)
+
+        for score in scores:
+            music = Node.void('music')
+            new.add_child(music)
+            music.add_child(Node.u32('music_id', score.id))
+            music.add_child(Node.u32('music_type', score.chart))
+            music.add_child(Node.u32('score', score.points))
+            music.add_child(Node.u32('cnt', score.plays))
+            music.add_child(Node.u32('combo', score.data.get_int('combo')))
+            music.add_child(Node.u32('clear_type', self.db_to_game_clear_type(score.data.get_int('clear_type'))))
+            music.add_child(Node.u32('score_grade', self.db_to_game_grade(score.data.get_int('grade'))))
+            stats = score.data.get_dict('stats')
+            music.add_child(Node.u32('btn_rate', stats.get_int('btn_rate')))
+            music.add_child(Node.u32('long_rate', stats.get_int('long_rate')))
+            music.add_child(Node.u32('vol_rate', stats.get_int('vol_rate')))
+
+        return game
+
+    def format_profile(self, userid: UserID, profile: ValidatedDict) -> Node:
+        game = Node.void('game_3')
+
+        # Generic profile stuff
+        game.add_child(Node.string('name', profile.get_str('name')))
+        game.add_child(Node.string('code', ID.format_extid(profile.get_int('extid'))))
+        game.add_child(Node.u32('gamecoin_packet', profile.get_int('packet')))
+        game.add_child(Node.u32('gamecoin_block', profile.get_int('block')))
+        game.add_child(Node.s16('skill_name_id', profile.get_int('skill_name_id', -1)))
+        game.add_child(Node.s32_array('hidden_param', profile.get_int_array('hidden_param', 20)))
+        game.add_child(Node.u32('blaster_energy', profile.get_int('blaster_energy')))
+        game.add_child(Node.u32('blaster_count', profile.get_int('blaster_count')))
+
+        # Enable Ryusei Festa
+        ryusei_festa = Node.void('ryusei_festa')
+        game.add_child(ryusei_festa)
+        ryusei_festa.add_child(Node.bool('ryusei_festa_trigger', True))
+
+        # Play statistics
+        statistics = self.get_play_statistics(userid)
+        last_play_date = statistics.get_int_array('last_play_date', 3)
+        today_play_date = Time.todays_date()
+        if (
+            last_play_date[0] == today_play_date[0] and
+            last_play_date[1] == today_play_date[1] and
+            last_play_date[2] == today_play_date[2]
+        ):
+            today_count = statistics.get_int('today_plays', 0)
+        else:
+            today_count = 0
+        game.add_child(Node.u32('play_count', statistics.get_int('total_plays', 0)))
+        game.add_child(Node.u32('daily_count', today_count))
+        game.add_child(Node.u32('play_chain', statistics.get_int('consecutive_days', 0)))
+
+        # Last played stuff
+        if 'last' in profile:
+            lastdict = profile.get_dict('last')
+            last = Node.void('last')
+            game.add_child(last)
+            last.add_child(Node.s32('music_id', lastdict.get_int('music_id', -1)))
+            last.add_child(Node.u8('music_type', lastdict.get_int('music_type')))
+            last.add_child(Node.u8('sort_type', lastdict.get_int('sort_type')))
+            last.add_child(Node.u8('narrow_down', lastdict.get_int('narrow_down')))
+            last.add_child(Node.u8('headphone', lastdict.get_int('headphone')))
+            last.add_child(Node.u16('appeal_id', lastdict.get_int('appeal_id', 1001)))
+            last.add_child(Node.u16('comment_id', lastdict.get_int('comment_id')))
+            last.add_child(Node.u8('gauge_option', lastdict.get_int('gauge_option')))
+
+        # Item unlocks
+        itemnode = Node.void('item')
+        game.add_child(itemnode)
+
+        game_config = self.get_game_config()
+        achievements = self.data.local.user.get_achievements(self.game, self.version, userid)
+
+        for item in achievements:
+            if item.type[:5] != 'item_':
+                continue
+            itemtype = int(item.type[5:])
+
+            if game_config.get_bool('force_unlock_songs') and itemtype == self.GAME_CATALOG_TYPE_SONG:
+                # Don't echo unlocked songs, we will add all of them later
+                continue
+
+            info = Node.void('info')
+            itemnode.add_child(info)
+            info.add_child(Node.u8('type', itemtype))
+            info.add_child(Node.u32('id', item.id))
+            info.add_child(Node.u32('param', item.data.get_int('param')))
+            if 'diff_param' in item.data:
+                info.add_child(Node.s32('diff_param', item.data.get_int('diff_param')))
+
+        if game_config.get_bool('force_unlock_songs'):
+            ids: Dict[int, int] = {}
+            songs = self.data.local.music.get_all_songs(self.game, self.version)
+            for song in songs:
+                if song.id not in ids:
+                    ids[song.id] = 0
+
+                if song.data.get_int('difficulty') > 0:
+                    ids[song.id] = ids[song.id] | (1 << song.chart)
+
+            for itemid in ids:
+                if ids[itemid] == 0:
+                    continue
+
+                info = Node.void('info')
+                itemnode.add_child(info)
+                info.add_child(Node.u8('type', self.GAME_CATALOG_TYPE_SONG))
+                info.add_child(Node.u32('id', itemid))
+                info.add_child(Node.u32('param', ids[itemid]))
+
+        return game
+
+    def unformat_profile(self, userid: UserID, request: Node, oldprofile: ValidatedDict) -> ValidatedDict:
+        newprofile = copy.deepcopy(oldprofile)
+
+        # Update blaster energy and in-game currencies
+        earned_gamecoin_packet = request.child_value('earned_gamecoin_packet')
+        if earned_gamecoin_packet is not None:
+            newprofile.replace_int('packet', newprofile.get_int('packet') + earned_gamecoin_packet)
+        earned_gamecoin_block = request.child_value('earned_gamecoin_block')
+        if earned_gamecoin_block is not None:
+            newprofile.replace_int('block', newprofile.get_int('block') + earned_gamecoin_block)
+        earned_blaster_energy = request.child_value('earned_blaster_energy')
+        if earned_blaster_energy is not None:
+            newprofile.replace_int('blaster_energy', newprofile.get_int('blaster_energy') + earned_blaster_energy)
+
+        # Miscelaneous stuff
+        newprofile.replace_int('blaster_count', request.child_value('blaster_count'))
+        newprofile.replace_int('skill_name_id', request.child_value('skill_name_id'))
+        newprofile.replace_int_array('hidden_param', 20, request.child_value('hidden_param'))
+
+        # Update user's unlock status if we aren't force unlocked
+        game_config = self.get_game_config()
+
+        if request.child('item') is not None:
+            for child in request.child('item').children:
+                if child.name != 'info':
+                    continue
+
+                item_id = child.child_value('id')
+                item_type = child.child_value('type')
+                param = child.child_value('param')
+                diff_param = child.child_value('diff_param')
+
+                if game_config.get_bool('force_unlock_songs') and item_type == self.GAME_CATALOG_TYPE_SONG:
+                    # Don't save back songs, because they were force unlocked
+                    continue
+
+                if diff_param is not None:
+                    paramvals = {
+                        'diff_param': diff_param,
+                        'param': param,
+                    }
+                else:
+                    paramvals = {
+                        'param': param,
+                    }
+
+                self.data.local.user.put_achievement(
+                    self.game,
+                    self.version,
+                    userid,
+                    item_id,
+                    'item_{}'.format(item_type),
+                    paramvals,
+                )
+
+        # Grab last information.
+        lastdict = newprofile.get_dict('last')
+        lastdict.replace_int('headphone', request.child_value('headphone'))
+        lastdict.replace_int('appeal_id', request.child_value('appeal_id'))
+        lastdict.replace_int('comment_id', request.child_value('comment_id'))
+        lastdict.replace_int('music_id', request.child_value('music_id'))
+        lastdict.replace_int('music_type', request.child_value('music_type'))
+        lastdict.replace_int('sort_type', request.child_value('sort_type'))
+        lastdict.replace_int('narrow_down', request.child_value('narrow_down'))
+        lastdict.replace_int('gauge_option', request.child_value('gauge_option'))
+
+        # Save back last information gleaned from results
+        newprofile.replace_dict('last', lastdict)
+
+        # Keep track of play statistics
+        self.update_play_statistics(userid)
+
+        return newprofile

bemani/backend/popn/__init__.py

@@ -0,0 +1,2 @@
+from bemani.backend.popn.factory import PopnMusicFactory
+from bemani.backend.popn.base import PopnMusicBase

bemani/backend/popn/base.py

@@ -0,0 +1,282 @@
+# vim: set fileencoding=utf-8
+from typing import Dict, Optional, Sequence
+
+from bemani.backend.base import Base
+from bemani.backend.core import CoreHandler, CardManagerHandler, PASELIHandler
+from bemani.common import ValidatedDict, Time, GameConstants, DBConstants
+from bemani.data import UserID, Achievement, ScoreSaveException
+from bemani.protocol import Node
+
+
+class PopnMusicBase(CoreHandler, CardManagerHandler, PASELIHandler, Base):
+    """
+    Base game class for all Pop'n Music versions. Handles common functionality for
+    getting profiles based on refid, creating new profiles, looking up and saving
+    scores.
+    """
+
+    game = GameConstants.POPN_MUSIC
+
+    # Play medals, as saved into/loaded from the DB
+    PLAY_MEDAL_CIRCLE_FAILED = DBConstants.POPN_MUSIC_PLAY_MEDAL_CIRCLE_FAILED
+    PLAY_MEDAL_DIAMOND_FAILED = DBConstants.POPN_MUSIC_PLAY_MEDAL_DIAMOND_FAILED
+    PLAY_MEDAL_STAR_FAILED = DBConstants.POPN_MUSIC_PLAY_MEDAL_STAR_FAILED
+    PLAY_MEDAL_EASY_CLEAR = DBConstants.POPN_MUSIC_PLAY_MEDAL_EASY_CLEAR
+    PLAY_MEDAL_CIRCLE_CLEARED = DBConstants.POPN_MUSIC_PLAY_MEDAL_CIRCLE_CLEARED
+    PLAY_MEDAL_DIAMOND_CLEARED = DBConstants.POPN_MUSIC_PLAY_MEDAL_DIAMOND_CLEARED
+    PLAY_MEDAL_STAR_CLEARED = DBConstants.POPN_MUSIC_PLAY_MEDAL_STAR_CLEARED
+    PLAY_MEDAL_CIRCLE_FULL_COMBO = DBConstants.POPN_MUSIC_PLAY_MEDAL_CIRCLE_FULL_COMBO
+    PLAY_MEDAL_DIAMOND_FULL_COMBO = DBConstants.POPN_MUSIC_PLAY_MEDAL_DIAMOND_FULL_COMBO
+    PLAY_MEDAL_STAR_FULL_COMBO = DBConstants.POPN_MUSIC_PLAY_MEDAL_STAR_FULL_COMBO
+    PLAY_MEDAL_PERFECT = DBConstants.POPN_MUSIC_PLAY_MEDAL_PERFECT
+
+    # Chart type, as saved into/loaded from the DB, and returned to game
+    CHART_TYPE_EASY = 0
+    CHART_TYPE_NORMAL = 1
+    CHART_TYPE_HYPER = 2
+    CHART_TYPE_EX = 3
+
+    # Old profile lookup type, for loading profile by ID
+    NEW_PROFILE_ONLY = 0
+    OLD_PROFILE_ONLY = 1
+    OLD_PROFILE_FALLTHROUGH = 2
+
+    def previous_version(self) -> Optional['PopnMusicBase']:
+        """
+        Returns the previous version of the game, based on this game. Should
+        be overridden.
+        """
+        return None
+
+    def format_profile(self, userid: UserID, profile: ValidatedDict) -> Node:
+        """
+        Base handler for a profile. Given a userid and a profile dictionary,
+        return a Node representing a profile. Should be overridden.
+        """
+        return Node.void('playerdata')
+
+    def format_conversion(self, userid: UserID, profile: ValidatedDict) -> Node:
+        """
+        Base handler for profile conversion. Given a userid and a profile
+        dictionary, return a node which represents the converted profile for
+        the next version of this game. Games will call previous_version to get
+        a game class of their previous game version, and then will call
+        format_conversion on that previous version to get the profile to
+        migrate.
+        """
+        return Node.void('playerdata')
+
+    def unformat_profile(self, userid: UserID, request: Node, oldprofile: ValidatedDict) -> ValidatedDict:
+        """
+        Base handler for profile parsing. Given a request and an old profile,
+        return a new profile that's been updated with the contents of the request.
+        Should be overridden.
+        """
+        return oldprofile
+
+    def get_profile_by_refid(self, refid: Optional[str], load_mode: int) -> Optional[Node]:
+        """
+        Given a RefID, return a formatted profile node. Basically every game
+        needs a profile lookup, even if it handles where that happens in
+        a different request. This is provided for code deduplication. This
+        method handles delegating to either format_profile, or looking up
+        the previous game and calling format_conversion, whenever necessary.
+        """
+        if refid is None:
+            return None
+
+        userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        if userid is None:
+            # User doesn't exist but should at this point
+            return None
+
+        if load_mode == self.OLD_PROFILE_ONLY:
+            # Trying to import from older version
+            oldversion = self.previous_version()
+            profile = oldversion.get_profile(userid)
+            if profile is None:
+                return None
+            return oldversion.format_conversion(userid, profile)
+        elif load_mode == self.NEW_PROFILE_ONLY:
+            # Trying to import from current version
+            profile = self.get_profile(userid)
+            if profile is None:
+                return None
+            return self.format_profile(userid, profile)
+        elif load_mode == self.OLD_PROFILE_FALLTHROUGH:
+            # Try to load from current, if that fails try to load previous
+            profile = self.get_profile(userid)
+            if profile is not None:
+                return self.format_profile(userid, profile)
+            oldversion = self.previous_version()
+            oldprofile = oldversion.get_profile(userid)
+            if oldprofile is not None:
+                return oldversion.format_conversion(userid, oldprofile)
+            return None
+        else:
+            # Unknown value
+            raise Exception("Unrecognized value for get profile!")
+
+    def new_profile_by_refid(
+        self,
+        refid: Optional[str],
+        name: Optional[str],
+        chara: Optional[int]=None,
+        achievements: Sequence[Achievement] = (),
+    ) -> Node:
+        """
+        Given a RefID and an optional name, create a profile and then return
+        a formatted profile node. Similar rationale to get_profile_by_refid.
+        """
+        if refid is None:
+            return None
+
+        if name is None:
+            name = 'なし'
+
+        userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        if userid is None:
+            raise Exception("Logic error! Didn't find user to tie profile to!")
+        defaultprofile = ValidatedDict({
+            'name': name,
+        })
+        if chara is not None:
+            defaultprofile.replace_int('chara', chara)
+        self.put_profile(userid, defaultprofile)
+        for achievement in achievements:
+            self.data.local.user.put_achievement(
+                self.game,
+                self.version,
+                userid,
+                achievement.id,
+                achievement.type,
+                achievement.data,
+            )
+
+        profile = self.get_profile(userid)
+        if profile is None:
+            raise Exception("Logic error! Didn't find profile after writing it!")
+        return self.format_profile(userid, profile)
+
+    def update_score(
+        self,
+        userid: UserID,
+        songid: int,
+        chart: int,
+        points: int,
+        medal: int,
+        combo: Optional[int]=None,
+        stats: Optional[Dict[str, int]]=None,
+    ) -> None:
+        """
+        Given various pieces of a score, update the user's high score and score
+        history in a controlled manner, so all games in Pop'n series can expect
+        the same attributes in a score. Note that the medals passed here are
+        expected to be converted from game identifier to our internal identifier,
+        so that any game in the series may convert them back. In this way, a song
+        played on Pop'n 22 that exists in Pop'n 19 will still have scores/medals
+        going back all versions.
+        """
+        # Range check medals
+        if medal not in [
+            self.PLAY_MEDAL_CIRCLE_FAILED,
+            self.PLAY_MEDAL_DIAMOND_FAILED,
+            self.PLAY_MEDAL_STAR_FAILED,
+            self.PLAY_MEDAL_EASY_CLEAR,
+            self.PLAY_MEDAL_CIRCLE_CLEARED,
+            self.PLAY_MEDAL_DIAMOND_CLEARED,
+            self.PLAY_MEDAL_STAR_CLEARED,
+            self.PLAY_MEDAL_CIRCLE_FULL_COMBO,
+            self.PLAY_MEDAL_DIAMOND_FULL_COMBO,
+            self.PLAY_MEDAL_STAR_FULL_COMBO,
+            self.PLAY_MEDAL_PERFECT,
+        ]:
+            raise Exception("Invalid medal value {}".format(medal))
+
+        oldscore = self.data.local.music.get_score(
+            self.game,
+            self.version,
+            userid,
+            songid,
+            chart,
+        )
+
+        # Score history is verbatum, instead of highest score
+        history = ValidatedDict({})
+        oldpoints = points
+
+        if oldscore is None:
+            # If it is a new score, create a new dictionary to add to
+            scoredata = ValidatedDict({})
+            raised = True
+            highscore = True
+        else:
+            # Set the score to any new record achieved
+            raised = points > oldscore.points
+            highscore = points >= oldscore.points
+            points = max(points, oldscore.points)
+            scoredata = oldscore.data
+
+        # Replace medal with highest value
+        scoredata.replace_int('medal', max(scoredata.get_int('medal'), medal))
+        history.replace_int('medal', medal)
+
+        if stats is not None:
+            if raised:
+                # We have stats, and there's a new high score, update the stats
+                scoredata.replace_dict('stats', stats)
+            history.replace_dict('stats', stats)
+
+        if combo is not None:
+            # If we have a combo, replace it
+            scoredata.replace_int('combo', max(scoredata.get_int('combo'), combo))
+            history.replace_int('combo', combo)
+
+        # Look up where this score was earned
+        lid = self.get_machine_id()
+
+        # Pop'n Music for all versions before Lapistoria sends all of the songs
+        # a player played at the end of the round. It doesn't send timestamps
+        # for those songs (Jubeat does). So, if a user plays the same song/chart
+        # more than once in a round, we will end up failing to store the attempt
+        # since we don't allow two of the same attempt at the same time for the
+        # same user and song/chart. So, bump the timestamp by one second and retry
+        # well past the maximum number of songs.
+        now = Time.now()
+        for bump in range(10):
+            timestamp = now + bump
+
+            # Write the new score back
+            self.data.local.music.put_score(
+                self.game,
+                self.version,
+                userid,
+                songid,
+                chart,
+                lid,
+                points,
+                scoredata,
+                highscore,
+                timestamp=timestamp,
+            )
+
+            try:
+                # Save the history of this score too
+                self.data.local.music.put_attempt(
+                    self.game,
+                    self.version,
+                    userid,
+                    songid,
+                    chart,
+                    lid,
+                    oldpoints,
+                    history,
+                    raised,
+                    timestamp=timestamp,
+                )
+            except ScoreSaveException:
+                # Try again one second in the future
+                continue
+
+            # We saved successfully
+            break

bemani/backend/popn/eclale.py

@@ -0,0 +1,737 @@
+# vim: set fileencoding=utf-8
+import binascii
+import copy
+from typing import Dict, Optional
+
+from bemani.backend.popn.base import PopnMusicBase
+from bemani.backend.popn.lapistoria import PopnMusicLapistoria
+
+from bemani.common import Time, ValidatedDict, VersionConstants
+from bemani.data import UserID
+from bemani.protocol import Node
+
+
+class PopnMusicEclale(PopnMusicBase):
+
+    name = "Pop'n Music éclale"
+    version = VersionConstants.POPN_MUSIC_ECLALE
+
+    # Chart type, as returned from the game
+    GAME_CHART_TYPE_EASY = 0
+    GAME_CHART_TYPE_NORMAL = 1
+    GAME_CHART_TYPE_HYPER = 2
+    GAME_CHART_TYPE_EX = 3
+
+    # Medal type, as returned from the game
+    GAME_PLAY_MEDAL_CIRCLE_FAILED = 1
+    GAME_PLAY_MEDAL_DIAMOND_FAILED = 2
+    GAME_PLAY_MEDAL_STAR_FAILED = 3
+    GAME_PLAY_MEDAL_EASY_CLEAR = 4
+    GAME_PLAY_MEDAL_CIRCLE_CLEARED = 5
+    GAME_PLAY_MEDAL_DIAMOND_CLEARED = 6
+    GAME_PLAY_MEDAL_STAR_CLEARED = 7
+    GAME_PLAY_MEDAL_CIRCLE_FULL_COMBO = 8
+    GAME_PLAY_MEDAL_DIAMOND_FULL_COMBO = 9
+    GAME_PLAY_MEDAL_STAR_FULL_COMBO = 10
+    GAME_PLAY_MEDAL_PERFECT = 11
+
+    # Biggest ID in the music DB
+    GAME_MAX_MUSIC_ID = 1550
+
+    def previous_version(self) -> Optional[PopnMusicBase]:
+        return PopnMusicLapistoria(self.data, self.config, self.model)
+
+    def __construct_common_info(self, root: Node) -> None:
+        # Event phases
+        phases = {
+            # Unknown event (0-16)
+            0: 16,
+            # Unknown event (0-3)
+            1: 3,
+            # Unknown event (0-1)
+            2: 1,
+            # Unknown event (0-2)
+            3: 2,
+            # Unknown event (0-1)
+            4: 1,
+            # Unknown event (0-2)
+            5: 2,
+            # Unknown event (0-1)
+            6: 1,
+            # Unknown event (0-4)
+            7: 4,
+            # Unknown event (0-3)
+            8: 3,
+            # Unknown event (0-4)
+            9: 4,
+            # Unknown event (0-4)
+            10: 4,
+            # Unknown event (0-1)
+            11: 1,
+            # Possibly global event matching related? (0-1)
+            12: 1,
+            # Unknown event (0-4)
+            13: 4,
+        }
+
+        for phaseid in phases:
+            phase = Node.void('phase')
+            root.add_child(phase)
+            phase.add_child(Node.s16('event_id', phaseid))
+            phase.add_child(Node.s16('phase', phases[phaseid]))
+
+        for areaid in range(1, 50):
+            area = Node.void('area')
+            root.add_child(area)
+            area.add_child(Node.s16('area_id', areaid))
+            area.add_child(Node.u64('end_date', 0))
+            area.add_child(Node.s16('medal_id', areaid))
+            area.add_child(Node.bool('is_limit', False))
+
+        # Calculate most popular characters
+        profiles = self.data.remote.user.get_all_profiles(self.game, self.version)
+        charas: Dict[int, int] = {}
+        for (userid, profile) in profiles:
+            chara = profile.get_int('chara', -1)
+            if chara <= 0:
+                continue
+            if chara not in charas:
+                charas[chara] = 1
+            else:
+                charas[chara] = charas[chara] + 1
+
+        # Order a typle by most popular character to least popular character
+        charamap = sorted(
+            [(c, charas[c]) for c in charas],
+            key=lambda c: c[1],
+            reverse=True,
+        )
+
+        # Output the top 20 of them
+        rank = 1
+        for (charaid, usecount) in charamap[:20]:
+            popular = Node.void('popular')
+            root.add_child(popular)
+            popular.add_child(Node.s16('rank', rank))
+            popular.add_child(Node.s16('chara_num', charaid))
+            rank = rank + 1
+
+        # Output the hit chart
+        for (songid, plays) in self.data.local.music.get_hit_chart(self.game, self.version, 500):
+            popular_music = Node.void('popular_music')
+            root.add_child(popular_music)
+            popular_music.add_child(Node.s16('music_num', songid))
+
+        # Output goods prices
+        for goodsid in range(1, 421):
+            if goodsid >= 1 and goodsid <= 80:
+                price = 60
+            elif goodsid >= 81 and goodsid <= 120:
+                price = 250
+            elif goodsid >= 121 and goodsid <= 142:
+                price = 500
+            elif goodsid >= 143 and goodsid <= 300:
+                price = 100
+            elif goodsid >= 301 and goodsid <= 420:
+                price = 150
+            else:
+                raise Exception('Invalid goods ID!')
+            goods = Node.void('goods')
+            root.add_child(goods)
+            goods.add_child(Node.s16('goods_id', goodsid))
+            goods.add_child(Node.s32('price', price))
+            goods.add_child(Node.s16('goods_type', 0))
+
+    def handle_pcb23_boot_request(self, request: Node) -> Node:
+        return Node.void('pcb23')
+
+    def handle_pcb23_error_request(self, request: Node) -> Node:
+        return Node.void('pcb23')
+
+    def handle_pcb23_dlstatus_request(self, request: Node) -> Node:
+        return Node.void('pcb23')
+
+    def handle_pcb23_write_request(self, request: Node) -> Node:
+        # Update the name of this cab for admin purposes
+        self.update_machine_name(request.child_value('pcb_setting/name'))
+        return Node.void('pcb23')
+
+    def handle_info23_common_request(self, request: Node) -> Node:
+        info = Node.void('info23')
+        self.__construct_common_info(info)
+        return info
+
+    def handle_lobby22_request(self, request: Node) -> Optional[Node]:
+        # Stub out the entire lobby22 service (yes, its lobby22 in Pop'n 23)
+        return Node.void('lobby22')
+
+    def handle_player23_start_request(self, request: Node) -> Node:
+        root = Node.void('player23')
+        root.add_child(Node.s32('play_id', 0))
+        self.__construct_common_info(root)
+        return root
+
+    def handle_player23_logout_request(self, request: Node) -> Node:
+        return Node.void('player23')
+
+    def handle_player23_read_request(self, request: Node) -> Node:
+        refid = request.child_value('ref_id')
+        root = self.get_profile_by_refid(refid, self.OLD_PROFILE_FALLTHROUGH)
+        if root is None:
+            root = Node.void('player23')
+            root.add_child(Node.s8('result', 2))
+        return root
+
+    def handle_player23_write_request(self, request: Node) -> Node:
+        refid = request.child_value('ref_id')
+
+        if refid is not None:
+            userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        else:
+            userid = None
+
+        if userid is not None:
+            oldprofile = self.get_profile(userid) or ValidatedDict()
+            newprofile = self.unformat_profile(userid, request, oldprofile)
+
+            if newprofile is not None:
+                self.put_profile(userid, newprofile)
+
+        return Node.void('player23')
+
+    def handle_player23_new_request(self, request: Node) -> Node:
+        refid = request.child_value('ref_id')
+        name = request.child_value('name')
+        root = self.new_profile_by_refid(refid, name)
+        if root is None:
+            root = Node.void('player23')
+            root.add_child(Node.s8('result', 2))
+        return root
+
+    def handle_player23_conversion_request(self, request: Node) -> Node:
+        refid = request.child_value('ref_id')
+        name = request.child_value('name')
+        chara = request.child_value('chara')
+        root = self.new_profile_by_refid(refid, name, chara)
+        if root is None:
+            root = Node.void('player23')
+            root.add_child(Node.s8('result', 2))
+        return root
+
+    def handle_player23_buy_request(self, request: Node) -> Node:
+        refid = request.child_value('ref_id')
+
+        if refid is not None:
+            userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        else:
+            userid = None
+
+        if userid is not None:
+            itemid = request.child_value('id')
+            itemtype = request.child_value('type')
+            itemparam = request.child_value('param')
+
+            price = request.child_value('price')
+            lumina = request.child_value('lumina')
+
+            if lumina >= price:
+                # Update player lumina balance
+                profile = self.get_profile(userid) or ValidatedDict()
+                profile.replace_int('lumina', lumina - price)
+                self.put_profile(userid, profile)
+
+                # Grant the object
+                self.data.local.user.put_achievement(
+                    self.game,
+                    self.version,
+                    userid,
+                    itemid,
+                    'item_{}'.format(itemtype),
+                    {
+                        'param': itemparam,
+                        'is_new': True,
+                    },
+                )
+
+        return Node.void('player23')
+
+    def handle_player23_read_score_request(self, request: Node) -> Node:
+        refid = request.child_value('ref_id')
+
+        root = Node.void('player23')
+
+        userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        if userid is not None:
+            scores = self.data.remote.music.get_scores(self.game, self.version, userid)
+        else:
+            scores = []
+
+        for score in scores:
+            # Skip any scores for chart types we don't support
+            if score.chart not in [
+                self.CHART_TYPE_EASY,
+                self.CHART_TYPE_NORMAL,
+                self.CHART_TYPE_HYPER,
+                self.CHART_TYPE_EX,
+            ]:
+                continue
+
+            points = score.points
+            medal = score.data.get_int('medal')
+
+            music = Node.void('music')
+            root.add_child(music)
+            music.add_child(Node.s16('music_num', score.id))
+            music.add_child(Node.u8('sheet_num', {
+                self.CHART_TYPE_EASY: self.GAME_CHART_TYPE_EASY,
+                self.CHART_TYPE_NORMAL: self.GAME_CHART_TYPE_NORMAL,
+                self.CHART_TYPE_HYPER: self.GAME_CHART_TYPE_HYPER,
+                self.CHART_TYPE_EX: self.GAME_CHART_TYPE_EX,
+            }[score.chart]))
+            music.add_child(Node.s32('score', points))
+            music.add_child(Node.u8('clear_type', {
+                self.PLAY_MEDAL_CIRCLE_FAILED: self.GAME_PLAY_MEDAL_CIRCLE_FAILED,
+                self.PLAY_MEDAL_DIAMOND_FAILED: self.GAME_PLAY_MEDAL_DIAMOND_FAILED,
+                self.PLAY_MEDAL_STAR_FAILED: self.GAME_PLAY_MEDAL_STAR_FAILED,
+                self.PLAY_MEDAL_EASY_CLEAR: self.GAME_PLAY_MEDAL_EASY_CLEAR,
+                self.PLAY_MEDAL_CIRCLE_CLEARED: self.GAME_PLAY_MEDAL_CIRCLE_CLEARED,
+                self.PLAY_MEDAL_DIAMOND_CLEARED: self.GAME_PLAY_MEDAL_DIAMOND_CLEARED,
+                self.PLAY_MEDAL_STAR_CLEARED: self.GAME_PLAY_MEDAL_STAR_CLEARED,
+                self.PLAY_MEDAL_CIRCLE_FULL_COMBO: self.GAME_PLAY_MEDAL_CIRCLE_FULL_COMBO,
+                self.PLAY_MEDAL_DIAMOND_FULL_COMBO: self.GAME_PLAY_MEDAL_DIAMOND_FULL_COMBO,
+                self.PLAY_MEDAL_STAR_FULL_COMBO: self.GAME_PLAY_MEDAL_STAR_FULL_COMBO,
+                self.PLAY_MEDAL_PERFECT: self.GAME_PLAY_MEDAL_PERFECT,
+            }[medal]))
+            music.add_child(Node.s16('cnt', score.plays))
+
+        return root
+
+    def handle_player23_write_music_request(self, request: Node) -> Node:
+        refid = request.child_value('ref_id')
+
+        root = Node.void('player23')
+        if refid is None:
+            return root
+
+        userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        if userid is None:
+            return root
+
+        songid = request.child_value('music_num')
+        chart = {
+            self.GAME_CHART_TYPE_EASY: self.CHART_TYPE_EASY,
+            self.GAME_CHART_TYPE_NORMAL: self.CHART_TYPE_NORMAL,
+            self.GAME_CHART_TYPE_HYPER: self.CHART_TYPE_HYPER,
+            self.GAME_CHART_TYPE_EX: self.CHART_TYPE_EX,
+        }[request.child_value('sheet_num')]
+        medal = request.child_value('clearmedal')
+        points = request.child_value('score')
+        combo = request.child_value('combo')
+        stats = {
+            'cool': request.child_value('cool'),
+            'great': request.child_value('great'),
+            'good': request.child_value('good'),
+            'bad': request.child_value('bad')
+        }
+        medal = {
+            self.GAME_PLAY_MEDAL_CIRCLE_FAILED: self.PLAY_MEDAL_CIRCLE_FAILED,
+            self.GAME_PLAY_MEDAL_DIAMOND_FAILED: self.PLAY_MEDAL_DIAMOND_FAILED,
+            self.GAME_PLAY_MEDAL_STAR_FAILED: self.PLAY_MEDAL_STAR_FAILED,
+            self.GAME_PLAY_MEDAL_EASY_CLEAR: self.PLAY_MEDAL_EASY_CLEAR,
+            self.GAME_PLAY_MEDAL_CIRCLE_CLEARED: self.PLAY_MEDAL_CIRCLE_CLEARED,
+            self.GAME_PLAY_MEDAL_DIAMOND_CLEARED: self.PLAY_MEDAL_DIAMOND_CLEARED,
+            self.GAME_PLAY_MEDAL_STAR_CLEARED: self.PLAY_MEDAL_STAR_CLEARED,
+            self.GAME_PLAY_MEDAL_CIRCLE_FULL_COMBO: self.PLAY_MEDAL_CIRCLE_FULL_COMBO,
+            self.GAME_PLAY_MEDAL_DIAMOND_FULL_COMBO: self.PLAY_MEDAL_DIAMOND_FULL_COMBO,
+            self.GAME_PLAY_MEDAL_STAR_FULL_COMBO: self.PLAY_MEDAL_STAR_FULL_COMBO,
+            self.GAME_PLAY_MEDAL_PERFECT: self.PLAY_MEDAL_PERFECT,
+        }[medal]
+        self.update_score(userid, songid, chart, points, medal, combo=combo, stats=stats)
+        return root
+
+    def format_conversion(self, userid: UserID, profile: ValidatedDict) -> Node:
+        root = Node.void('player23')
+        root.add_child(Node.string('name', profile.get_str('name', 'なし')))
+        root.add_child(Node.s16('chara', profile.get_int('chara', -1)))
+        root.add_child(Node.s8('result', 1))
+
+        scores = self.data.remote.music.get_scores(self.game, self.version, userid)
+        for score in scores:
+            # Skip any scores for chart types we don't support
+            if score.chart not in [
+                self.CHART_TYPE_EASY,
+                self.CHART_TYPE_NORMAL,
+                self.CHART_TYPE_HYPER,
+                self.CHART_TYPE_EX,
+            ]:
+                continue
+
+            points = score.points
+            medal = score.data.get_int('medal')
+
+            music = Node.void('music')
+            root.add_child(music)
+            music.add_child(Node.s16('music_num', score.id))
+            music.add_child(Node.u8('sheet_num', {
+                self.CHART_TYPE_EASY: self.GAME_CHART_TYPE_EASY,
+                self.CHART_TYPE_NORMAL: self.GAME_CHART_TYPE_NORMAL,
+                self.CHART_TYPE_HYPER: self.GAME_CHART_TYPE_HYPER,
+                self.CHART_TYPE_EX: self.GAME_CHART_TYPE_EX,
+            }[score.chart]))
+            music.add_child(Node.s32('score', points))
+            music.add_child(Node.u8('clear_type', {
+                self.PLAY_MEDAL_CIRCLE_FAILED: self.GAME_PLAY_MEDAL_CIRCLE_FAILED,
+                self.PLAY_MEDAL_DIAMOND_FAILED: self.GAME_PLAY_MEDAL_DIAMOND_FAILED,
+                self.PLAY_MEDAL_STAR_FAILED: self.GAME_PLAY_MEDAL_STAR_FAILED,
+                self.PLAY_MEDAL_EASY_CLEAR: self.GAME_PLAY_MEDAL_EASY_CLEAR,
+                self.PLAY_MEDAL_CIRCLE_CLEARED: self.GAME_PLAY_MEDAL_CIRCLE_CLEARED,
+                self.PLAY_MEDAL_DIAMOND_CLEARED: self.GAME_PLAY_MEDAL_DIAMOND_CLEARED,
+                self.PLAY_MEDAL_STAR_CLEARED: self.GAME_PLAY_MEDAL_STAR_CLEARED,
+                self.PLAY_MEDAL_CIRCLE_FULL_COMBO: self.GAME_PLAY_MEDAL_CIRCLE_FULL_COMBO,
+                self.PLAY_MEDAL_DIAMOND_FULL_COMBO: self.GAME_PLAY_MEDAL_DIAMOND_FULL_COMBO,
+                self.PLAY_MEDAL_STAR_FULL_COMBO: self.GAME_PLAY_MEDAL_STAR_FULL_COMBO,
+                self.PLAY_MEDAL_PERFECT: self.GAME_PLAY_MEDAL_PERFECT,
+            }[medal]))
+            music.add_child(Node.s16('cnt', score.plays))
+
+        return root
+
+    def format_extid(self, extid: int) -> str:
+        data = str(extid)
+        crc = abs(binascii.crc32(data.encode('ascii'))) % 10000
+        return '{}{:04d}'.format(data, crc)
+
+    def format_profile(self, userid: UserID, profile: ValidatedDict) -> Node:
+        root = Node.void('player23')
+
+        # Mark this as a current profile
+        root.add_child(Node.s8('result', 0))
+
+        # Account stuff
+        account = Node.void('account')
+        root.add_child(account)
+        account.add_child(Node.string('g_pm_id', self.format_extid(profile.get_int('extid'))))  # Eclale formats on its own
+        account.add_child(Node.string('name', profile.get_str('name', 'なし')))
+        account.add_child(Node.s8('tutorial', profile.get_int('tutorial')))
+        account.add_child(Node.s16('area_id', profile.get_int('area_id')))
+        account.add_child(Node.s16('lumina', profile.get_int('lumina', 300)))
+        account.add_child(Node.s16('read_news', profile.get_int('read_news')))
+        account.add_child(Node.bool('welcom_pack', profile.get_bool('welcom_pack')))
+        account.add_child(Node.s16_array('medal_set', profile.get_int_array('medal_set', 4)))
+        account.add_child(Node.s16_array('nice', profile.get_int_array('nice', 30, [-1] * 30)))
+        account.add_child(Node.s16_array('favorite_chara', profile.get_int_array('favorite_chara', 20, [-1] * 20)))
+        account.add_child(Node.s16_array('special_area', profile.get_int_array('special_area', 8)))
+        account.add_child(Node.s16_array('chocolate_charalist', profile.get_int_array('chocolate_charalist', 5, [-1] * 5)))
+        account.add_child(Node.s16_array('teacher_setting', profile.get_int_array('teacher_setting', 10)))
+
+        # Stuff we never change
+        account.add_child(Node.s8('staff', 0))
+        account.add_child(Node.s16('item_type', 0))
+        account.add_child(Node.s16('item_id', 0))
+        account.add_child(Node.s8('is_conv', 0))
+        account.add_child(Node.bool('meteor_flg', True))
+        account.add_child(Node.s16_array('license_data', [-1] * 20))
+
+        # Add statistics section
+        last_played = [x[0] for x in self.data.local.music.get_last_played(self.game, self.version, userid, 5)]
+        most_played = [x[0] for x in self.data.local.music.get_most_played(self.game, self.version, userid, 10)]
+        while len(last_played) < 5:
+            last_played.append(-1)
+        while len(most_played) < 10:
+            most_played.append(-1)
+
+        account.add_child(Node.s16_array('my_best', most_played))
+        account.add_child(Node.s16_array('latest_music', last_played))
+
+        # TODO: Hook up rivals for Pop'n music?
+        account.add_child(Node.u8('active_fr_num', 0))
+
+        # player statistics
+        statistics = self.get_play_statistics(userid)
+        last_play_date = statistics.get_int_array('last_play_date', 3)
+        today_play_date = Time.todays_date()
+        if (
+            last_play_date[0] == today_play_date[0] and
+            last_play_date[1] == today_play_date[1] and
+            last_play_date[2] == today_play_date[2]
+        ):
+            today_count = statistics.get_int('today_plays', 0)
+        else:
+            today_count = 0
+        account.add_child(Node.s16('total_play_cnt', statistics.get_int('total_plays', 0)))
+        account.add_child(Node.s16('today_play_cnt', today_count))
+        account.add_child(Node.s16('consecutive_days', statistics.get_int('consecutive_days', 0)))
+        account.add_child(Node.s16('total_days', statistics.get_int('total_days', 0)))
+        account.add_child(Node.s16('interval_day', 0))
+
+        # Set up info node
+        info = Node.void('info')
+        root.add_child(info)
+        info.add_child(Node.u16('ep', profile.get_int('ep')))
+
+        # Set up last information
+        config = Node.void('config')
+        root.add_child(config)
+        config.add_child(Node.u8('mode', profile.get_int('mode')))
+        config.add_child(Node.s16('chara', profile.get_int('chara', -1)))
+        config.add_child(Node.s16('music', profile.get_int('music', -1)))
+        config.add_child(Node.u8('sheet', profile.get_int('sheet')))
+        config.add_child(Node.s8('category', profile.get_int('category', -1)))
+        config.add_child(Node.s8('sub_category', profile.get_int('sub_category', -1)))
+        config.add_child(Node.s8('chara_category', profile.get_int('chara_category', -1)))
+        config.add_child(Node.s16('course_id', profile.get_int('course_id', -1)))
+        config.add_child(Node.s8('course_folder', profile.get_int('course_folder', -1)))
+        config.add_child(Node.s8('ms_banner_disp', profile.get_int('ms_banner_disp')))
+        config.add_child(Node.s8('ms_down_info', profile.get_int('ms_down_info')))
+        config.add_child(Node.s8('ms_side_info', profile.get_int('ms_side_info')))
+        config.add_child(Node.s8('ms_raise_type', profile.get_int('ms_raise_type')))
+        config.add_child(Node.s8('ms_rnd_type', profile.get_int('ms_rnd_type')))
+
+        # Player options
+        option = Node.void('option')
+        option_dict = profile.get_dict('option')
+        root.add_child(option)
+        option.add_child(Node.s16('hispeed', option_dict.get_int('hispeed')))
+        option.add_child(Node.u8('popkun', option_dict.get_int('popkun')))
+        option.add_child(Node.bool('hidden', option_dict.get_bool('hidden')))
+        option.add_child(Node.s16('hidden_rate', option_dict.get_int('hidden_rate')))
+        option.add_child(Node.bool('sudden', option_dict.get_bool('sudden')))
+        option.add_child(Node.s16('sudden_rate', option_dict.get_int('sudden_rate')))
+        option.add_child(Node.s8('randmir', option_dict.get_int('randmir')))
+        option.add_child(Node.s8('gauge_type', option_dict.get_int('gauge_type')))
+        option.add_child(Node.u8('ojama_0', option_dict.get_int('ojama_0')))
+        option.add_child(Node.u8('ojama_1', option_dict.get_int('ojama_1')))
+        option.add_child(Node.bool('forever_0', option_dict.get_bool('forever_0')))
+        option.add_child(Node.bool('forever_1', option_dict.get_bool('forever_1')))
+        option.add_child(Node.bool('full_setting', option_dict.get_bool('full_setting')))
+        option.add_child(Node.u8('judge', option_dict.get_int('judge')))
+
+        # Unknown custom category stuff?
+        custom_cate = Node.void('custom_cate')
+        root.add_child(custom_cate)
+        custom_cate.add_child(Node.s8('valid', 0))
+        custom_cate.add_child(Node.s8('lv_min', -1))
+        custom_cate.add_child(Node.s8('lv_max', -1))
+        custom_cate.add_child(Node.s8('medal_min', -1))
+        custom_cate.add_child(Node.s8('medal_max', -1))
+        custom_cate.add_child(Node.s8('friend_no', -1))
+        custom_cate.add_child(Node.s8('score_flg', -1))
+
+        # Set up achievements
+        achievements = self.data.local.user.get_achievements(self.game, self.version, userid)
+        for achievement in achievements:
+            if achievement.type[:5] == 'item_':
+                itemtype = int(achievement.type[5:])
+                param = achievement.data.get_int('param')
+                is_new = achievement.data.get_bool('is_new')
+
+                item = Node.void('item')
+                root.add_child(item)
+                # Type is the type of unlock/item. Type 0 is song unlock in Eclale.
+                # In this case, the id is the song ID according to the game. Unclear
+                # what the param is supposed to be, but i've seen 8 and 0. Might be
+                # what chart is available?
+                item.add_child(Node.u8('type', itemtype))
+                item.add_child(Node.u16('id', achievement.id))
+                item.add_child(Node.u16('param', param))
+                item.add_child(Node.bool('is_new', is_new))
+
+            elif achievement.type == 'chara':
+                friendship = achievement.data.get_int('friendship')
+
+                chara = Node.void('chara_param')
+                root.add_child(chara)
+                chara.add_child(Node.u16('chara_id', achievement.id))
+                chara.add_child(Node.u16('friendship', friendship))
+
+            elif achievement.type == 'medal':
+                level = achievement.data.get_int('level')
+                exp = achievement.data.get_int('exp')
+                set_count = achievement.data.get_int('set_count')
+                get_count = achievement.data.get_int('get_count')
+
+                medal = Node.void('medal')
+                root.add_child(medal)
+                medal.add_child(Node.s16('medal_id', achievement.id))
+                medal.add_child(Node.s16('level', level))
+                medal.add_child(Node.s32('exp', exp))
+                medal.add_child(Node.s32('set_count', set_count))
+                medal.add_child(Node.s32('get_count', get_count))
+
+        # Unknown customizations
+        customize = Node.void('customize')
+        root.add_child(customize)
+        customize.add_child(Node.u16('effect_left', profile.get_int('effect_left')))
+        customize.add_child(Node.u16('effect_center', profile.get_int('effect_center')))
+        customize.add_child(Node.u16('effect_right', profile.get_int('effect_right')))
+        customize.add_child(Node.u16('hukidashi', profile.get_int('hukidashi')))
+        customize.add_child(Node.u16('comment_1', profile.get_int('comment_1')))
+        customize.add_child(Node.u16('comment_2', profile.get_int('comment_2')))
+
+        # NetVS section
+        netvs = Node.void('netvs')
+        root.add_child(netvs)
+        netvs.add_child(Node.s16_array('record', [0] * 6))
+        netvs.add_child(Node.string('dialog', ''))
+        netvs.add_child(Node.string('dialog', ''))
+        netvs.add_child(Node.string('dialog', ''))
+        netvs.add_child(Node.string('dialog', ''))
+        netvs.add_child(Node.string('dialog', ''))
+        netvs.add_child(Node.string('dialog', ''))
+        netvs.add_child(Node.s8_array('ojama_condition', [0] * 74))
+        netvs.add_child(Node.s8_array('set_ojama', [0] * 3))
+        netvs.add_child(Node.s8_array('set_recommend', [0] * 3))
+        netvs.add_child(Node.u32('netvs_play_cnt', 0))
+
+        # Event stuff
+        event = Node.void('event')
+        root.add_child(event)
+        event.add_child(Node.s16('enemy_medal', profile.get_int('event_enemy_medal')))
+        event.add_child(Node.s16('hp', profile.get_int('event_hp')))
+
+        # Stamp stuff
+        stamp = Node.void('stamp')
+        root.add_child(stamp)
+        stamp.add_child(Node.s16('stamp_id', profile.get_int('stamp_id')))
+        stamp.add_child(Node.s16('cnt', profile.get_int('stamp_cnt')))
+
+        return root
+
+    def unformat_profile(self, userid: UserID, request: Node, oldprofile: ValidatedDict) -> ValidatedDict:
+        newprofile = copy.deepcopy(oldprofile)
+
+        # Set that we've seen this profile
+        newprofile.replace_bool('welcom_pack', True)
+
+        account = request.child('account')
+        if account is not None:
+            newprofile.replace_int('tutorial', account.child_value('tutorial'))
+            newprofile.replace_int('read_news', account.child_value('read_news'))
+            newprofile.replace_int('area_id', account.child_value('area_id'))
+            newprofile.replace_int('lumina', account.child_value('lumina'))
+            newprofile.replace_int_array('medal_set', 4, account.child_value('medal_set'))
+            newprofile.replace_int_array('nice', 30, account.child_value('nice'))
+            newprofile.replace_int_array('favorite_chara', 20, account.child_value('favorite_chara'))
+            newprofile.replace_int_array('special_area', 8, account.child_value('special_area'))
+            newprofile.replace_int_array('chocolate_charalist', 5, account.child_value('chocolate_charalist'))
+            newprofile.replace_int_array('teacher_setting', 10, account.child_value('teacher_setting'))
+
+        info = request.child('info')
+        if info is not None:
+            newprofile.replace_int('ep', info.child_value('ep'))
+
+        config = request.child('config')
+        if config is not None:
+            newprofile.replace_int('mode', config.child_value('mode'))
+            newprofile.replace_int('chara', config.child_value('chara'))
+            newprofile.replace_int('music', config.child_value('music'))
+            newprofile.replace_int('sheet', config.child_value('sheet'))
+            newprofile.replace_int('category', config.child_value('category'))
+            newprofile.replace_int('sub_category', config.child_value('sub_category'))
+            newprofile.replace_int('chara_category', config.child_value('chara_category'))
+            newprofile.replace_int('course_id', config.child_value('course_id'))
+            newprofile.replace_int('course_folder', config.child_value('course_folder'))
+            newprofile.replace_int('ms_banner_disp', config.child_value('ms_banner_disp'))
+            newprofile.replace_int('ms_down_info', config.child_value('ms_down_info'))
+            newprofile.replace_int('ms_side_info', config.child_value('ms_side_info'))
+            newprofile.replace_int('ms_raise_type', config.child_value('ms_raise_type'))
+            newprofile.replace_int('ms_rnd_type', config.child_value('ms_rnd_type'))
+
+        option_dict = newprofile.get_dict('option')
+        option = request.child('option')
+        if option is not None:
+            option_dict.replace_int('hispeed', option.child_value('hispeed'))
+            option_dict.replace_int('popkun', option.child_value('popkun'))
+            option_dict.replace_bool('hidden', option.child_value('hidden'))
+            option_dict.replace_int('hidden_rate', option.child_value('hidden_rate'))
+            option_dict.replace_bool('sudden', option.child_value('sudden'))
+            option_dict.replace_int('sudden_rate', option.child_value('sudden_rate'))
+            option_dict.replace_int('randmir', option.child_value('randmir'))
+            option_dict.replace_int('gauge_type', option.child_value('gauge_type'))
+            option_dict.replace_int('ojama_0', option.child_value('ojama_0'))
+            option_dict.replace_int('ojama_1', option.child_value('ojama_1'))
+            option_dict.replace_bool('forever_0', option.child_value('forever_0'))
+            option_dict.replace_bool('forever_1', option.child_value('forever_1'))
+            option_dict.replace_bool('full_setting', option.child_value('full_setting'))
+            option_dict.replace_int('judge', option.child_value('judge'))
+        newprofile.replace_dict('option', option_dict)
+
+        customize = request.child('customize')
+        if customize is not None:
+            newprofile.replace_int('effect_left', customize.child_value('effect_left'))
+            newprofile.replace_int('effect_center', customize.child_value('effect_center'))
+            newprofile.replace_int('effect_right', customize.child_value('effect_right'))
+            newprofile.replace_int('hukidashi', customize.child_value('hukidashi'))
+            newprofile.replace_int('comment_1', customize.child_value('comment_1'))
+            newprofile.replace_int('comment_2', customize.child_value('comment_2'))
+
+        event = request.child('event')
+        if event is not None:
+            newprofile.replace_int('event_enemy_medal', event.child_value('enemy_medal'))
+            newprofile.replace_int('event_hp', event.child_value('hp'))
+
+        stamp = request.child('stamp')
+        if stamp is not None:
+            newprofile.replace_int('stamp_id', stamp.child_value('stamp_id'))
+            newprofile.replace_int('stamp_cnt', stamp.child_value('cnt'))
+
+        # Extract achievements
+        for node in request.children:
+            if node.name == 'item':
+                itemid = node.child_value('id')
+                itemtype = node.child_value('type')
+                param = node.child_value('param')
+                is_new = node.child_value('is_new')
+
+                self.data.local.user.put_achievement(
+                    self.game,
+                    self.version,
+                    userid,
+                    itemid,
+                    'item_{}'.format(itemtype),
+                    {
+                        'param': param,
+                        'is_new': is_new,
+                    },
+                )
+
+            elif node.name == 'chara_param':
+                charaid = node.child_value('chara_id')
+                friendship = node.child_value('friendship')
+
+                self.data.local.user.put_achievement(
+                    self.game,
+                    self.version,
+                    userid,
+                    charaid,
+                    'chara',
+                    {
+                        'friendship': friendship,
+                    },
+                )
+
+            elif node.name == 'medal':
+                medalid = node.child_value('medal_id')
+                level = node.child_value('level')
+                exp = node.child_value('exp')
+                set_count = node.child_value('set_count')
+                get_count = node.child_value('get_count')
+
+                self.data.local.user.put_achievement(
+                    self.game,
+                    self.version,
+                    userid,
+                    medalid,
+                    'medal',
+                    {
+                        'level': level,
+                        'exp': exp,
+                        'set_count': set_count,
+                        'get_count': get_count,
+                    },
+                )
+
+        # Keep track of play statistics
+        self.update_play_statistics(userid)
+
+        return newprofile

bemani/backend/popn/factory.py

@@ -0,0 +1,133 @@
+from typing import Dict, Optional, Any
+
+from bemani.backend.base import Base, Factory
+from bemani.backend.popn.stubs import (
+    PopnMusic,
+    PopnMusic2,
+    PopnMusic3,
+    PopnMusic4,
+    PopnMusic5,
+    PopnMusic6,
+    PopnMusic7,
+    PopnMusic8,
+    PopnMusic9,
+    PopnMusic10,
+    PopnMusic11,
+    PopnMusicIroha,
+    PopnMusicCarnival,
+    PopnMusicFever,
+    PopnMusicAdventure,
+    PopnMusicParty,
+    PopnMusicTheMovie,
+    PopnMusicSengokuRetsuden,
+)
+from bemani.backend.popn.tunestreet import PopnMusicTuneStreet
+from bemani.backend.popn.fantasia import PopnMusicFantasia
+from bemani.backend.popn.sunnypark import PopnMusicSunnyPark
+from bemani.backend.popn.lapistoria import PopnMusicLapistoria
+from bemani.backend.popn.eclale import PopnMusicEclale
+from bemani.backend.popn.usaneko import PopnMusicUsaNeko
+from bemani.backend.popn.peace import PopnMusicPeace
+from bemani.common import Model, VersionConstants
+from bemani.data import Data
+
+
+class PopnMusicFactory(Factory):
+
+    MANAGED_CLASSES = [
+        PopnMusic,
+        PopnMusic2,
+        PopnMusic3,
+        PopnMusic4,
+        PopnMusic5,
+        PopnMusic6,
+        PopnMusic7,
+        PopnMusic8,
+        PopnMusic9,
+        PopnMusic10,
+        PopnMusic11,
+        PopnMusicIroha,
+        PopnMusicCarnival,
+        PopnMusicFever,
+        PopnMusicAdventure,
+        PopnMusicParty,
+        PopnMusicTheMovie,
+        PopnMusicSengokuRetsuden,
+        PopnMusicTuneStreet,
+        PopnMusicFantasia,
+        PopnMusicSunnyPark,
+        PopnMusicLapistoria,
+        PopnMusicEclale,
+        PopnMusicUsaNeko,
+        PopnMusicPeace,
+    ]
+
+    @classmethod
+    def register_all(cls) -> None:
+        for game in ['G15', 'H16', 'I17', 'J39', 'K39', 'L39', 'M39']:
+            Base.register(game, PopnMusicFactory)
+
+    @classmethod
+    def create(cls, data: Data, config: Dict[str, Any], model: Model, parentmodel: Optional[Model]=None) -> Optional[Base]:
+
+        def version_from_date(date: int) -> Optional[int]:
+            if date <= 2014061900:
+                return VersionConstants.POPN_MUSIC_SUNNY_PARK
+            if date >= 2014062500 and date < 2015112600:
+                return VersionConstants.POPN_MUSIC_LAPISTORIA
+            if date >= 2015112600 and date < 2016121400:
+                return VersionConstants.POPN_MUSIC_ECLALE
+            if date >= 2016121400 and date < 2018101700:
+                return VersionConstants.POPN_MUSIC_USANEKO
+            if date >= 2018101700:
+                return VersionConstants.POPN_MUSIC_PEACE
+            return None
+
+        if model.game == 'G15':
+            return PopnMusicAdventure(data, config, model)
+        if model.game == 'H16':
+            return PopnMusicParty(data, config, model)
+        if model.game == 'I17':
+            return PopnMusicTheMovie(data, config, model)
+        if model.game == 'J39':
+            return PopnMusicSengokuRetsuden(data, config, model)
+        if model.game == 'K39':
+            return PopnMusicTuneStreet(data, config, model)
+        if model.game == 'L39':
+            return PopnMusicFantasia(data, config, model)
+        if model.game == 'M39':
+            if model.version is None:
+                if parentmodel is None:
+                    return None
+
+                # We have no way to tell apart newer versions. However, we can make
+                # an educated guess if we happen to be summoned for old profile lookup.
+                if parentmodel.game not in ['G15', 'H16', 'I17', 'J39', 'K39', 'L39', 'M39']:
+                    return None
+                parentversion = version_from_date(parentmodel.version)
+                if parentversion == VersionConstants.POPN_MUSIC_LAPISTORIA:
+                    return PopnMusicSunnyPark(data, config, model)
+                if parentversion == VersionConstants.POPN_MUSIC_ECLALE:
+                    return PopnMusicLapistoria(data, config, model)
+                if parentversion == VersionConstants.POPN_MUSIC_USANEKO:
+                    return PopnMusicEclale(data, config, model)
+                if parentversion == VersionConstants.POPN_MUSIC_PEACE:
+                    return PopnMusicUsaNeko(data, config, model)
+
+                # Unknown older version
+                return None
+
+            version = version_from_date(model.version)
+            if version == VersionConstants.POPN_MUSIC_SUNNY_PARK:
+                return PopnMusicSunnyPark(data, config, model)
+            if version == VersionConstants.POPN_MUSIC_LAPISTORIA:
+                return PopnMusicLapistoria(data, config, model)
+            if version == VersionConstants.POPN_MUSIC_ECLALE:
+                return PopnMusicEclale(data, config, model)
+            if version == VersionConstants.POPN_MUSIC_USANEKO:
+                return PopnMusicUsaNeko(data, config, model)
+            if version == VersionConstants.POPN_MUSIC_PEACE:
+                return PopnMusicPeace(data, config, model)
+
+        # Unknown game version
+        return None

bemani/backend/popn/fantasia.py

@@ -0,0 +1,460 @@
+# vim: set fileencoding=utf-8
+import copy
+from typing import Optional
+
+from bemani.backend.popn.base import PopnMusicBase
+from bemani.backend.popn.tunestreet import PopnMusicTuneStreet
+
+from bemani.backend.base import Status
+from bemani.common import ValidatedDict, VersionConstants, Time, ID
+from bemani.data import Score, UserID
+from bemani.protocol import Node
+
+
+class PopnMusicFantasia(PopnMusicBase):
+
+    name = "Pop'n Music fantasia"
+    version = VersionConstants.POPN_MUSIC_FANTASIA
+
+    # Chart type, as returned from the game
+    GAME_CHART_TYPE_EASY = 2
+    GAME_CHART_TYPE_NORMAL = 0
+    GAME_CHART_TYPE_HYPER = 1
+    GAME_CHART_TYPE_EX = 3
+
+    # Chart type, as packed into a hiscore binary
+    GAME_CHART_TYPE_EASY_POSITION = 0
+    GAME_CHART_TYPE_NORMAL_POSITION = 1
+    GAME_CHART_TYPE_HYPER_POSITION = 2
+    GAME_CHART_TYPE_EX_POSITION = 3
+
+    # Medal type, as returned from the game
+    GAME_PLAY_MEDAL_CIRCLE_FAILED = 1
+    GAME_PLAY_MEDAL_DIAMOND_FAILED = 2
+    GAME_PLAY_MEDAL_STAR_FAILED = 3
+    GAME_PLAY_MEDAL_CIRCLE_CLEARED = 5
+    GAME_PLAY_MEDAL_DIAMOND_CLEARED = 6
+    GAME_PLAY_MEDAL_STAR_CLEARED = 7
+    GAME_PLAY_MEDAL_CIRCLE_FULL_COMBO = 9
+    GAME_PLAY_MEDAL_DIAMOND_FULL_COMBO = 10
+    GAME_PLAY_MEDAL_STAR_FULL_COMBO = 11
+    GAME_PLAY_MEDAL_PERFECT = 15
+
+    # Maximum music ID for this game
+    GAME_MAX_MUSIC_ID = 1150
+
+    def previous_version(self) -> Optional[PopnMusicBase]:
+        return PopnMusicTuneStreet(self.data, self.config, self.model)
+
+    def __format_medal_for_score(self, score: Score) -> int:
+        medal = {
+            self.PLAY_MEDAL_CIRCLE_FAILED: self.GAME_PLAY_MEDAL_CIRCLE_FAILED,
+            self.PLAY_MEDAL_DIAMOND_FAILED: self.GAME_PLAY_MEDAL_DIAMOND_FAILED,
+            self.PLAY_MEDAL_STAR_FAILED: self.GAME_PLAY_MEDAL_STAR_FAILED,
+            self.PLAY_MEDAL_EASY_CLEAR: self.GAME_PLAY_MEDAL_CIRCLE_CLEARED,  # Map approximately
+            self.PLAY_MEDAL_CIRCLE_CLEARED: self.GAME_PLAY_MEDAL_CIRCLE_CLEARED,
+            self.PLAY_MEDAL_DIAMOND_CLEARED: self.GAME_PLAY_MEDAL_DIAMOND_CLEARED,
+            self.PLAY_MEDAL_STAR_CLEARED: self.GAME_PLAY_MEDAL_STAR_CLEARED,
+            self.PLAY_MEDAL_CIRCLE_FULL_COMBO: self.GAME_PLAY_MEDAL_CIRCLE_FULL_COMBO,
+            self.PLAY_MEDAL_DIAMOND_FULL_COMBO: self.GAME_PLAY_MEDAL_DIAMOND_FULL_COMBO,
+            self.PLAY_MEDAL_STAR_FULL_COMBO: self.GAME_PLAY_MEDAL_STAR_FULL_COMBO,
+            self.PLAY_MEDAL_PERFECT: self.GAME_PLAY_MEDAL_PERFECT,
+        }[score.data.get_int('medal')]
+        position = {
+            self.CHART_TYPE_EASY: self.GAME_CHART_TYPE_EASY_POSITION,
+            self.CHART_TYPE_NORMAL: self.GAME_CHART_TYPE_NORMAL_POSITION,
+            self.CHART_TYPE_HYPER: self.GAME_CHART_TYPE_HYPER_POSITION,
+            self.CHART_TYPE_EX: self.GAME_CHART_TYPE_EX_POSITION,
+        }[score.chart]
+        return medal << (position * 4)
+
+    def format_profile(self, userid: UserID, profile: ValidatedDict) -> Node:
+        root = Node.void('playerdata')
+
+        # Set up the base profile
+        base = Node.void('base')
+        root.add_child(base)
+        base.add_child(Node.string('name', profile.get_str('name', 'なし')))
+        base.add_child(Node.string('g_pm_id', ID.format_extid(profile.get_int('extid'))))
+        base.add_child(Node.u8('mode', profile.get_int('mode', 0)))
+        base.add_child(Node.s8('button', profile.get_int('button', 0)))
+        base.add_child(Node.s8('last_play_flag', profile.get_int('last_play_flag', -1)))
+        base.add_child(Node.u8('medal_and_friend', profile.get_int('medal_and_friend', 0)))
+        base.add_child(Node.s8('category', profile.get_int('category', -1)))
+        base.add_child(Node.s8('sub_category', profile.get_int('sub_category', -1)))
+        base.add_child(Node.s16('chara', profile.get_int('chara', -1)))
+        base.add_child(Node.s8('chara_category', profile.get_int('chara_category', -1)))
+        base.add_child(Node.u8('collabo', profile.get_int('collabo', 255)))
+        base.add_child(Node.u8('sheet', profile.get_int('sheet', 0)))
+        base.add_child(Node.s8('tutorial', profile.get_int('tutorial', 0)))
+        base.add_child(Node.s32('music_open_pt', profile.get_int('music_open_pt', 0)))
+        base.add_child(Node.s8('is_conv', -1))
+        base.add_child(Node.s32('option', profile.get_int('option', 0)))
+        base.add_child(Node.s16('music', profile.get_int('music', -1)))
+        base.add_child(Node.u16('ep', profile.get_int('ep', 0)))
+        base.add_child(Node.s32_array('sp_color_flg', profile.get_int_array('sp_color_flg', 2)))
+        base.add_child(Node.s32('read_news', profile.get_int('read_news', 0)))
+        base.add_child(Node.s16('consecutive_days_coupon', profile.get_int('consecutive_days_coupon', 0)))
+        base.add_child(Node.s8('staff', 0))
+
+        # Player card section
+        player_card_dict = profile.get_dict('player_card')
+        player_card = Node.void('player_card')
+        root.add_child(player_card)
+        player_card.add_child(Node.u8_array('title', player_card_dict.get_int_array('title', 2, [0, 1])))
+        player_card.add_child(Node.u8('frame', player_card_dict.get_int('frame')))
+        player_card.add_child(Node.u8('base', player_card_dict.get_int('base')))
+        player_card.add_child(Node.u8_array('seal', player_card_dict.get_int_array('seal', 2)))
+        player_card.add_child(Node.s32_array('get_title', player_card_dict.get_int_array('get_title', 4)))
+        player_card.add_child(Node.s32('get_frame', player_card_dict.get_int('get_frame')))
+        player_card.add_child(Node.s32('get_base', player_card_dict.get_int('get_base')))
+        player_card.add_child(Node.s32_array('get_seal', player_card_dict.get_int_array('get_seal', 2)))
+
+        # Player card EX section
+        player_card_ex = Node.void('player_card_ex')
+        root.add_child(player_card_ex)
+        player_card_ex.add_child(Node.s32('get_title_ex', player_card_dict.get_int('get_title_ex')))
+        player_card_ex.add_child(Node.s32('get_frame_ex', player_card_dict.get_int('get_frame_ex')))
+        player_card_ex.add_child(Node.s32('get_base_ex', player_card_dict.get_int('get_base_ex')))
+        player_card_ex.add_child(Node.s32('get_seal_ex', player_card_dict.get_int('get_seal_ex')))
+
+        # Statistics section and scores section
+        statistics = self.get_play_statistics(userid)
+        last_play_date = statistics.get_int_array('last_play_date', 3)
+        today_play_date = Time.todays_date()
+        if (
+            last_play_date[0] == today_play_date[0] and
+            last_play_date[1] == today_play_date[1] and
+            last_play_date[2] == today_play_date[2]
+        ):
+            today_count = statistics.get_int('today_plays', 0)
+        else:
+            today_count = 0
+        base.add_child(Node.u8('active_fr_num', 0))  # TODO: Hook up rivals code?
+        base.add_child(Node.s32('total_play_cnt', statistics.get_int('total_plays', 0)))
+        base.add_child(Node.s16('today_play_cnt', today_count))
+        base.add_child(Node.s16('consecutive_days', statistics.get_int('consecutive_days', 0)))
+
+        last_played = [x[0] for x in self.data.local.music.get_last_played(self.game, self.version, userid, 3)]
+        most_played = [x[0] for x in self.data.local.music.get_most_played(self.game, self.version, userid, 20)]
+        while len(last_played) < 3:
+            last_played.append(-1)
+        while len(most_played) < 20:
+            most_played.append(-1)
+
+        hiscore_array = [0] * int((((self.GAME_MAX_MUSIC_ID * 4) * 17) + 7) / 8)
+        clear_medal = [0] * self.GAME_MAX_MUSIC_ID
+        clear_medal_sub = [0] * self.GAME_MAX_MUSIC_ID
+
+        scores = self.data.remote.music.get_scores(self.game, self.version, userid)
+        for score in scores:
+            if score.id > self.GAME_MAX_MUSIC_ID:
+                continue
+
+            # Skip any scores for chart types we don't support
+            if score.chart not in [
+                self.CHART_TYPE_EASY,
+                self.CHART_TYPE_NORMAL,
+                self.CHART_TYPE_HYPER,
+                self.CHART_TYPE_EX,
+            ]:
+                continue
+
+            points = score.points
+            clear_medal[score.id] = clear_medal[score.id] | self.__format_medal_for_score(score)
+
+            hiscore_index = (score.id * 4) + {
+                self.CHART_TYPE_EASY: self.GAME_CHART_TYPE_EASY_POSITION,
+                self.CHART_TYPE_NORMAL: self.GAME_CHART_TYPE_NORMAL_POSITION,
+                self.CHART_TYPE_HYPER: self.GAME_CHART_TYPE_HYPER_POSITION,
+                self.CHART_TYPE_EX: self.GAME_CHART_TYPE_EX_POSITION,
+            }[score.chart]
+            hiscore_byte_pos = int((hiscore_index * 17) / 8)
+            hiscore_bit_pos = int((hiscore_index * 17) % 8)
+            hiscore_value = points << hiscore_bit_pos
+            hiscore_array[hiscore_byte_pos] = hiscore_array[hiscore_byte_pos] | (hiscore_value & 0xFF)
+            hiscore_array[hiscore_byte_pos + 1] = hiscore_array[hiscore_byte_pos + 1] | ((hiscore_value >> 8) & 0xFF)
+            hiscore_array[hiscore_byte_pos + 2] = hiscore_array[hiscore_byte_pos + 2] | ((hiscore_value >> 16) & 0xFF)
+
+        hiscore = bytes(hiscore_array)
+
+        player_card.add_child(Node.s16_array('best_music', most_played[0:3]))
+        base.add_child(Node.s16_array('my_best', most_played))
+        base.add_child(Node.s16_array('latest_music', last_played))
+        base.add_child(Node.u16_array('clear_medal', clear_medal))
+        base.add_child(Node.u8_array('clear_medal_sub', clear_medal_sub))
+
+        # Goes outside of base for some reason
+        root.add_child(Node.binary('hiscore', hiscore))
+
+        # Net VS section
+        netvs = Node.void('netvs')
+        root.add_child(netvs)
+        netvs.add_child(Node.s32_array('get_ojama', [0, 0]))
+        netvs.add_child(Node.s32('rank_point', 0))
+        netvs.add_child(Node.s32('play_point', 0))
+        netvs.add_child(Node.s16_array('record', [0, 0, 0, 0, 0, 0]))
+        netvs.add_child(Node.u8('rank', 0))
+        netvs.add_child(Node.s8_array('ojama_condition', [0] * 74))
+        netvs.add_child(Node.s8_array('set_ojama', [0, 0, 0]))
+        netvs.add_child(Node.s8_array('set_recommend', [0, 0, 0]))
+        netvs.add_child(Node.s8_array('jewelry', [0] * 15))
+        for dialog in [0, 1, 2, 3, 4, 5]:
+            # TODO: Configure this, maybe?
+            netvs.add_child(Node.string('dialog', 'dialog#{}'.format(dialog)))
+
+        sp_data = Node.void('sp_data')
+        root.add_child(sp_data)
+        sp_data.add_child(Node.s32('sp', profile.get_int('sp', 0)))
+
+        reflec_data = Node.void('reflec_data')
+        root.add_child(reflec_data)
+        reflec_data.add_child(Node.s8_array('reflec', profile.get_int_array('reflec', 2)))
+
+        # Navigate section
+        navigate_dict = profile.get_dict('navigate')
+        navigate = Node.void('navigate')
+        root.add_child(navigate)
+        navigate.add_child(Node.s8('genre', navigate_dict.get_int('genre')))
+        navigate.add_child(Node.s8('image', navigate_dict.get_int('image')))
+        navigate.add_child(Node.s8('level', navigate_dict.get_int('level')))
+        navigate.add_child(Node.s8('ojama', navigate_dict.get_int('ojama')))
+        navigate.add_child(Node.s16('limit_num', navigate_dict.get_int('limit_num')))
+        navigate.add_child(Node.s8('button', navigate_dict.get_int('button')))
+        navigate.add_child(Node.s8('life', navigate_dict.get_int('life')))
+        navigate.add_child(Node.s16('progress', navigate_dict.get_int('progress')))
+
+        return root
+
+    def format_conversion(self, userid: UserID, profile: ValidatedDict) -> Node:
+        root = Node.void('playerdata')
+
+        root.add_child(Node.string('name', profile.get_str('name', 'なし')))
+        root.add_child(Node.s16('chara', profile.get_int('chara', -1)))
+        root.add_child(Node.s32('option', profile.get_int('option', 0)))
+        root.add_child(Node.u8('version', 0))
+        root.add_child(Node.u8('kind', 0))
+        root.add_child(Node.u8('season', 0))
+
+        clear_medal = [0] * self.GAME_MAX_MUSIC_ID
+
+        scores = self.data.remote.music.get_scores(self.game, self.version, userid)
+        for score in scores:
+            if score.id > self.GAME_MAX_MUSIC_ID:
+                continue
+
+            # Skip any scores for chart types we don't support
+            if score.chart not in [
+                self.CHART_TYPE_EASY,
+                self.CHART_TYPE_NORMAL,
+                self.CHART_TYPE_HYPER,
+                self.CHART_TYPE_EX,
+            ]:
+                continue
+
+            clear_medal[score.id] = clear_medal[score.id] | self.__format_medal_for_score(score)
+
+        root.add_child(Node.u16_array('clear_medal', clear_medal))
+
+        return root
+
+    def unformat_profile(self, userid: UserID, request: Node, oldprofile: ValidatedDict) -> ValidatedDict:
+        # For some reason, Pop'n 20 sends us two profile saves, one with 'not done yet'
+        # so we only want to process the done yet node. The 'not gameover' save has
+        # jubeat collabo stuff set in it, but we don't use that so it doesn't matter.
+        if request.child_value('is_not_gameover') == 1:
+            return oldprofile
+
+        newprofile = copy.deepcopy(oldprofile)
+        newprofile.replace_int('option', request.child_value('option'))
+        newprofile.replace_int('chara', request.child_value('chara'))
+        newprofile.replace_int('mode', request.child_value('mode'))
+        newprofile.replace_int('button', request.child_value('button'))
+        newprofile.replace_int('music', request.child_value('music'))
+        newprofile.replace_int('sheet', request.child_value('sheet'))
+        newprofile.replace_int('last_play_flag', request.child_value('last_play_flag'))
+        newprofile.replace_int('category', request.child_value('category'))
+        newprofile.replace_int('sub_category', request.child_value('sub_category'))
+        newprofile.replace_int('chara_category', request.child_value('chara_category'))
+        newprofile.replace_int('medal_and_friend', request.child_value('medal_and_friend'))
+        newprofile.replace_int('ep', request.child_value('ep'))
+        newprofile.replace_int_array('sp_color_flg', 2, request.child_value('sp_color_flg'))
+        newprofile.replace_int('read_news', request.child_value('read_news'))
+        newprofile.replace_int('consecutive_days_coupon', request.child_value('consecutive_days_coupon'))
+        newprofile.replace_int('tutorial', request.child_value('tutorial'))
+        newprofile.replace_int('music_open_pt', request.child_value('music_open_pt'))
+        newprofile.replace_int('collabo', request.child_value('collabo'))
+
+        sp_node = request.child('sp_data')
+        if sp_node is not None:
+            newprofile.replace_int('sp', sp_node.child_value('sp'))
+
+        reflec_node = request.child('reflec_data')
+        if reflec_node is not None:
+            newprofile.replace_int_array('reflec', 2, reflec_node.child_value('reflec'))
+
+        # Keep track of play statistics
+        self.update_play_statistics(userid)
+
+        # Extract player card stuff
+        player_card_dict = newprofile.get_dict('player_card')
+        player_card_dict.replace_int_array('title', 2, request.child_value('title'))
+        player_card_dict.replace_int('frame', request.child_value('frame'))
+        player_card_dict.replace_int('base', request.child_value('base'))
+        player_card_dict.replace_int_array('seal', 2, request.child_value('seal'))
+        player_card_dict.replace_int_array('get_title', 4, request.child_value('get_title'))
+        player_card_dict.replace_int('get_frame', request.child_value('get_frame'))
+        player_card_dict.replace_int('get_base', request.child_value('get_base'))
+        player_card_dict.replace_int_array('get_seal', 2, request.child_value('get_seal'))
+
+        player_card_ex = request.child('player_card_ex')
+        if player_card_ex is not None:
+            player_card_dict.replace_int('get_title_ex', player_card_ex.child_value('get_title_ex'))
+            player_card_dict.replace_int('get_frame_ex', player_card_ex.child_value('get_frame_ex'))
+            player_card_dict.replace_int('get_base_ex', player_card_ex.child_value('get_base_ex'))
+            player_card_dict.replace_int('get_seal_ex', player_card_ex.child_value('get_seal_ex'))
+        newprofile.replace_dict('player_card', player_card_dict)
+
+        # Extract navigate stuff
+        navigate_dict = newprofile.get_dict('navigate')
+        navigate = request.child('navigate')
+        if navigate is not None:
+            navigate_dict.replace_int('genre', navigate.child_value('genre'))
+            navigate_dict.replace_int('image', navigate.child_value('image'))
+            navigate_dict.replace_int('level', navigate.child_value('level'))
+            navigate_dict.replace_int('ojama', navigate.child_value('ojama'))
+            navigate_dict.replace_int('limit_num', navigate.child_value('limit_num'))
+            navigate_dict.replace_int('button', navigate.child_value('button'))
+            navigate_dict.replace_int('life', navigate.child_value('life'))
+            navigate_dict.replace_int('progress', navigate.child_value('progress'))
+        newprofile.replace_dict('navigate', navigate_dict)
+
+        # Extract scores
+        for node in request.children:
+            if node.name == 'stage':
+                songid = node.child_value('no')
+                chart = {
+                    self.GAME_CHART_TYPE_EASY: self.CHART_TYPE_EASY,
+                    self.GAME_CHART_TYPE_NORMAL: self.CHART_TYPE_NORMAL,
+                    self.GAME_CHART_TYPE_HYPER: self.CHART_TYPE_HYPER,
+                    self.GAME_CHART_TYPE_EX: self.CHART_TYPE_EX,
+                }[node.child_value('sheet')]
+                medal = (node.child_value('n_data') >> (chart * 4)) & 0x000F
+                medal = {
+                    self.GAME_PLAY_MEDAL_CIRCLE_FAILED: self.PLAY_MEDAL_CIRCLE_FAILED,
+                    self.GAME_PLAY_MEDAL_DIAMOND_FAILED: self.PLAY_MEDAL_DIAMOND_FAILED,
+                    self.GAME_PLAY_MEDAL_STAR_FAILED: self.PLAY_MEDAL_STAR_FAILED,
+                    self.GAME_PLAY_MEDAL_CIRCLE_CLEARED: self.PLAY_MEDAL_CIRCLE_CLEARED,
+                    self.GAME_PLAY_MEDAL_DIAMOND_CLEARED: self.PLAY_MEDAL_DIAMOND_CLEARED,
+                    self.GAME_PLAY_MEDAL_STAR_CLEARED: self.PLAY_MEDAL_STAR_CLEARED,
+                    self.GAME_PLAY_MEDAL_CIRCLE_FULL_COMBO: self.PLAY_MEDAL_CIRCLE_FULL_COMBO,
+                    self.GAME_PLAY_MEDAL_DIAMOND_FULL_COMBO: self.PLAY_MEDAL_DIAMOND_FULL_COMBO,
+                    self.GAME_PLAY_MEDAL_STAR_FULL_COMBO: self.PLAY_MEDAL_STAR_FULL_COMBO,
+                    self.GAME_PLAY_MEDAL_PERFECT: self.PLAY_MEDAL_PERFECT,
+                }[medal]
+                points = node.child_value('score')
+                self.update_score(userid, songid, chart, points, medal)
+
+        return newprofile
+
+    def handle_playerdata_request(self, request: Node) -> Optional[Node]:
+        method = request.attribute('method')
+
+        if method == 'expire':
+            return Node.void('playerdata')
+
+        elif method == 'logout':
+            return Node.void('playerdata')
+
+        elif method == 'get':
+            modelstring = request.attribute('model')
+            refid = request.child_value('ref_id')
+            root = self.get_profile_by_refid(
+                refid,
+                self.NEW_PROFILE_ONLY if modelstring is None else self.OLD_PROFILE_ONLY,
+            )
+            if root is None:
+                root = Node.void('playerdata')
+                root.set_attribute('status', str(Status.NO_PROFILE))
+            return root
+
+        elif method == 'conversion':
+            refid = request.child_value('ref_id')
+            name = request.child_value('name')
+            chara = request.child_value('chara')
+            root = self.new_profile_by_refid(refid, name, chara)
+            if root is None:
+                root = Node.void('playerdata')
+                root.set_attribute('status', str(Status.NO_PROFILE))
+            return root
+
+        elif method == 'new':
+            refid = request.child_value('ref_id')
+            name = request.child_value('name')
+            root = self.new_profile_by_refid(refid, name)
+            if root is None:
+                root = Node.void('playerdata')
+                root.set_attribute('status', str(Status.NO_PROFILE))
+            return root
+
+        elif method == 'set':
+            refid = request.attribute('ref_id')
+
+            root = Node.void('playerdata')
+            root.add_child(Node.s8('pref', -1))
+            if refid is None:
+                return root
+
+            userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+            if userid is None:
+                return root
+
+            oldprofile = self.get_profile(userid) or ValidatedDict()
+            newprofile = self.unformat_profile(userid, request, oldprofile)
+
+            if newprofile is not None:
+                self.put_profile(userid, newprofile)
+                root.add_child(Node.string('name', newprofile['name']))
+
+            return root
+
+        # Invalid method
+        return None
+
+    def handle_game_request(self, request: Node) -> Optional[Node]:
+        method = request.attribute('method')
+
+        if method == 'get':
+            # TODO: Hook these up to config so we can change this
+            root = Node.void('game')
+            root.add_child(Node.s32('game_phase', 2))
+            root.add_child(Node.s32('ir_phase', 0))
+            root.add_child(Node.s32('event_phase', 5))
+            root.add_child(Node.s32('netvs_phase', 0))
+            root.add_child(Node.s32('card_phase', 6))
+            root.add_child(Node.s32('illust_phase', 2))
+            root.add_child(Node.s32('psp_phase', 5))
+            root.add_child(Node.s32('other_phase', 1))
+            root.add_child(Node.s32('jubeat_phase', 1))
+            root.add_child(Node.s32('public_phase', 3))
+            root.add_child(Node.s32('kac_phase', 2))
+            root.add_child(Node.s32('local_matching', 1))
+            root.add_child(Node.s32('n_matching_sec', 60))
+            root.add_child(Node.s32('l_matching_sec', 60))
+            root.add_child(Node.s32('is_check_cpu', 0))
+            root.add_child(Node.s32('week_no', 0))
+            root.add_child(Node.s32_array('ng_illust', [0] * 10))
+            root.add_child(Node.s16_array('sel_ranking', [-1] * 10))
+            root.add_child(Node.s16_array('up_ranking', [-1] * 10))
+            return root
+
+        if method == 'active':
+            # Update the name of this cab for admin purposes
+            self.update_machine_name(request.child_value('shop_name'))
+            return Node.void('game')
+
+        if method == 'taxphase':
+            return Node.void('game')
+
+        # Invalid method
+        return None

bemani/backend/popn/lapistoria.py

@@ -0,0 +1,641 @@
+# vim: set fileencoding=utf-8
+import copy
+from typing import Optional
+
+from bemani.backend.popn.base import PopnMusicBase
+from bemani.backend.popn.sunnypark import PopnMusicSunnyPark
+
+from bemani.backend.base import Status
+from bemani.common import ValidatedDict, VersionConstants, Time, ID
+from bemani.data import UserID
+from bemani.protocol import Node
+
+
+class PopnMusicLapistoria(PopnMusicBase):
+
+    name = "Pop'n Music ラピストリア"
+    version = VersionConstants.POPN_MUSIC_LAPISTORIA
+
+    # Chart type, as returned from the game
+    GAME_CHART_TYPE_EASY = 0
+    GAME_CHART_TYPE_NORMAL = 1
+    GAME_CHART_TYPE_HYPER = 2
+    GAME_CHART_TYPE_EX = 3
+
+    # Medal type, as returned from the game
+    GAME_PLAY_MEDAL_CIRCLE_FAILED = 1
+    GAME_PLAY_MEDAL_DIAMOND_FAILED = 2
+    GAME_PLAY_MEDAL_STAR_FAILED = 3
+    GAME_PLAY_MEDAL_EASY_CLEAR = 4
+    GAME_PLAY_MEDAL_CIRCLE_CLEARED = 5
+    GAME_PLAY_MEDAL_DIAMOND_CLEARED = 6
+    GAME_PLAY_MEDAL_STAR_CLEARED = 7
+    GAME_PLAY_MEDAL_CIRCLE_FULL_COMBO = 8
+    GAME_PLAY_MEDAL_DIAMOND_FULL_COMBO = 9
+    GAME_PLAY_MEDAL_STAR_FULL_COMBO = 10
+    GAME_PLAY_MEDAL_PERFECT = 11
+
+    # Max valud music ID for conversions and stuff
+    GAME_MAX_MUSIC_ID = 1422
+
+    def previous_version(self) -> Optional[PopnMusicBase]:
+        return PopnMusicSunnyPark(self.data, self.config, self.model)
+
+    def handle_info22_request(self, request: Node) -> Optional[Node]:
+        method = request.attribute('method')
+
+        if method == 'common':
+            # TODO: Hook these up to config so we can change this
+            phases = {
+                # Unknown event
+                0: 0,
+                # Unknown event
+                1: 0,
+                # Pop'n Aura, max 10 (remov all aura requirements)
+                2: 10,
+                # Story
+                3: 1,
+                # BEMANI ruins Discovery!
+                4: 0,
+                # Unknown event
+                5: 0,
+                # Unknown event
+                6: 0,
+                # Unknown event
+                7: 0,
+                # Unknown event
+                8: 0,
+                # Unknown event
+                9: 0,
+                # Unknown event
+                10: 0,
+                # Unknown event
+                11: 0,
+                # Unknown event
+                12: 0,
+                # Unknown event
+                13: 0,
+                # Unknown event
+                14: 0,
+                # Unknown event
+                15: 0,
+                # Unknown event
+                16: 0,
+                # Unknown event
+                17: 0,
+                # Unknown event
+                18: 0,
+                # Unknown event
+                19: 0,
+            }
+            stories = list(range(173))
+
+            root = Node.void('info22')
+            for phaseid in phases:
+                phase = Node.void('phase')
+                root.add_child(phase)
+                phase.add_child(Node.s16('event_id', phaseid))
+                phase.add_child(Node.s16('phase', phases[phaseid]))
+
+            for storyid in stories:
+                story = Node.void('story')
+                root.add_child(story)
+                story.add_child(Node.u32('story_id', storyid))
+                story.add_child(Node.bool('is_limited', False))
+                story.add_child(Node.u64('limit_date', 0))
+
+            return root
+
+        # Invalid method
+        return None
+
+    def handle_pcb22_request(self, request: Node) -> Optional[Node]:
+        method = request.attribute('method')
+
+        if method == 'boot':
+            return Node.void('pcb22')
+        elif method == 'error':
+            return Node.void('pcb22')
+        elif method == 'write':
+            # Update the name of this cab for admin purposes
+            self.update_machine_name(request.child_value('pcb_setting/name'))
+            return Node.void('pcb22')
+
+        # Invalid method
+        return None
+
+    def handle_lobby22_request(self, request: Node) -> Optional[Node]:
+        # Stub out the entire lobby22 service
+        return Node.void('lobby22')
+
+    def handle_player22_request(self, request: Node) -> Optional[Node]:
+        method = request.attribute('method')
+
+        if method == 'read':
+            refid = request.child_value('ref_id')
+            # Pop'n Music 22 doesn't send a modelstring to load old profiles,
+            # it just expects us to know. So always look for old profiles in
+            # Pop'n 22 land.
+            root = self.get_profile_by_refid(refid, self.OLD_PROFILE_FALLTHROUGH)
+            if root is None:
+                root = Node.void('player22')
+                root.set_attribute('status', str(Status.NO_PROFILE))
+            return root
+
+        elif method == 'new':
+            refid = request.child_value('ref_id')
+            name = request.child_value('name')
+            root = self.new_profile_by_refid(refid, name)
+            if root is None:
+                root = Node.void('player22')
+                root.set_attribute('status', str(Status.NO_PROFILE))
+            return root
+
+        elif method == 'start':
+            return Node.void('player22')
+
+        elif method == 'logout':
+            return Node.void('player22')
+
+        elif method == 'write':
+            refid = request.child_value('ref_id')
+
+            root = Node.void('player22')
+            if refid is None:
+                return root
+
+            userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+            if userid is None:
+                return root
+
+            oldprofile = self.get_profile(userid) or ValidatedDict()
+            newprofile = self.unformat_profile(userid, request, oldprofile)
+
+            if newprofile is not None:
+                self.put_profile(userid, newprofile)
+
+            return root
+
+        elif method == 'conversion':
+            refid = request.child_value('ref_id')
+            name = request.child_value('name')
+            chara = request.child_value('chara')
+            root = self.new_profile_by_refid(refid, name, chara)
+            if root is None:
+                root = Node.void('playerdata')
+                root.set_attribute('status', str(Status.NO_PROFILE))
+            return root
+
+        elif method == 'write_music':
+            refid = request.child_value('ref_id')
+
+            root = Node.void('player22')
+            if refid is None:
+                return root
+
+            userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+            if userid is None:
+                return root
+
+            songid = request.child_value('music_num')
+            chart = {
+                self.GAME_CHART_TYPE_EASY: self.CHART_TYPE_EASY,
+                self.GAME_CHART_TYPE_NORMAL: self.CHART_TYPE_NORMAL,
+                self.GAME_CHART_TYPE_HYPER: self.CHART_TYPE_HYPER,
+                self.GAME_CHART_TYPE_EX: self.CHART_TYPE_EX,
+            }[request.child_value('sheet_num')]
+            medal = request.child_value('clearmedal')
+            points = request.child_value('score')
+            combo = request.child_value('combo')
+            stats = {
+                'cool': request.child_value('cool'),
+                'great': request.child_value('great'),
+                'good': request.child_value('good'),
+                'bad': request.child_value('bad')
+            }
+            medal = {
+                self.GAME_PLAY_MEDAL_CIRCLE_FAILED: self.PLAY_MEDAL_CIRCLE_FAILED,
+                self.GAME_PLAY_MEDAL_DIAMOND_FAILED: self.PLAY_MEDAL_DIAMOND_FAILED,
+                self.GAME_PLAY_MEDAL_STAR_FAILED: self.PLAY_MEDAL_STAR_FAILED,
+                self.GAME_PLAY_MEDAL_EASY_CLEAR: self.PLAY_MEDAL_EASY_CLEAR,
+                self.GAME_PLAY_MEDAL_CIRCLE_CLEARED: self.PLAY_MEDAL_CIRCLE_CLEARED,
+                self.GAME_PLAY_MEDAL_DIAMOND_CLEARED: self.PLAY_MEDAL_DIAMOND_CLEARED,
+                self.GAME_PLAY_MEDAL_STAR_CLEARED: self.PLAY_MEDAL_STAR_CLEARED,
+                self.GAME_PLAY_MEDAL_CIRCLE_FULL_COMBO: self.PLAY_MEDAL_CIRCLE_FULL_COMBO,
+                self.GAME_PLAY_MEDAL_DIAMOND_FULL_COMBO: self.PLAY_MEDAL_DIAMOND_FULL_COMBO,
+                self.GAME_PLAY_MEDAL_STAR_FULL_COMBO: self.PLAY_MEDAL_STAR_FULL_COMBO,
+                self.GAME_PLAY_MEDAL_PERFECT: self.PLAY_MEDAL_PERFECT,
+            }[medal]
+            self.update_score(userid, songid, chart, points, medal, combo=combo, stats=stats)
+            return root
+
+        # Invalid method
+        return None
+
+    def format_profile(self, userid: UserID, profile: ValidatedDict) -> Node:
+        root = Node.void('player22')
+
+        # Result
+        root.add_child(Node.s8('result', 0))
+
+        # Set up account
+        account = Node.void('account')
+        root.add_child(account)
+        account.add_child(Node.string('name', profile.get_str('name', 'なし')))
+        account.add_child(Node.string('g_pm_id', ID.format_extid(profile.get_int('extid'))))
+        account.add_child(Node.s8('tutorial', profile.get_int('tutorial', -1)))
+        account.add_child(Node.s16('read_news', profile.get_int('read_news', 0)))
+        account.add_child(Node.s8('staff', 0))
+        account.add_child(Node.s8('is_conv', 0))
+        account.add_child(Node.s16('item_type', 0))
+        account.add_child(Node.s16('item_id', 0))
+        account.add_child(Node.s16_array('license_data', [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1]))
+
+        # Statistics section and scores section
+        statistics = self.get_play_statistics(userid)
+        last_play_date = statistics.get_int_array('last_play_date', 3)
+        today_play_date = Time.todays_date()
+        if (
+            last_play_date[0] == today_play_date[0] and
+            last_play_date[1] == today_play_date[1] and
+            last_play_date[2] == today_play_date[2]
+        ):
+            today_count = statistics.get_int('today_plays', 0)
+        else:
+            today_count = 0
+        account.add_child(Node.u8('active_fr_num', 0))  # TODO: Hook up rivals code?
+        account.add_child(Node.s16('total_play_cnt', statistics.get_int('total_plays', 0)))
+        account.add_child(Node.s16('today_play_cnt', today_count))
+        account.add_child(Node.s16('consecutive_days', statistics.get_int('consecutive_days', 0)))
+        account.add_child(Node.s16('total_days', statistics.get_int('total_days', 0)))
+        account.add_child(Node.s16('interval_day', 0))
+
+        # Add scores section
+        last_played = [x[0] for x in self.data.local.music.get_last_played(self.game, self.version, userid, 5)]
+        most_played = [x[0] for x in self.data.local.music.get_most_played(self.game, self.version, userid, 10)]
+        while len(last_played) < 5:
+            last_played.append(-1)
+        while len(most_played) < 10:
+            most_played.append(-1)
+
+        account.add_child(Node.s16_array('my_best', most_played))
+        account.add_child(Node.s16_array('latest_music', last_played))
+
+        scores = self.data.remote.music.get_scores(self.game, self.version, userid)
+        for score in scores:
+            # Skip any scores for chart types we don't support
+            if score.chart not in [
+                self.CHART_TYPE_EASY,
+                self.CHART_TYPE_NORMAL,
+                self.CHART_TYPE_HYPER,
+                self.CHART_TYPE_EX,
+            ]:
+                continue
+
+            points = score.points
+            medal = score.data.get_int('medal')
+
+            music = Node.void('music')
+            root.add_child(music)
+            music.add_child(Node.s16('music_num', score.id))
+            music.add_child(Node.u8('sheet_num', {
+                self.CHART_TYPE_EASY: self.GAME_CHART_TYPE_EASY,
+                self.CHART_TYPE_NORMAL: self.GAME_CHART_TYPE_NORMAL,
+                self.CHART_TYPE_HYPER: self.GAME_CHART_TYPE_HYPER,
+                self.CHART_TYPE_EX: self.GAME_CHART_TYPE_EX,
+            }[score.chart]))
+            music.add_child(Node.s16('cnt', score.plays))
+            music.add_child(Node.s32('score', points))
+            music.add_child(Node.u8('clear_type', {
+                self.PLAY_MEDAL_CIRCLE_FAILED: self.GAME_PLAY_MEDAL_CIRCLE_FAILED,
+                self.PLAY_MEDAL_DIAMOND_FAILED: self.GAME_PLAY_MEDAL_DIAMOND_FAILED,
+                self.PLAY_MEDAL_STAR_FAILED: self.GAME_PLAY_MEDAL_STAR_FAILED,
+                self.PLAY_MEDAL_EASY_CLEAR: self.GAME_PLAY_MEDAL_EASY_CLEAR,
+                self.PLAY_MEDAL_CIRCLE_CLEARED: self.GAME_PLAY_MEDAL_CIRCLE_CLEARED,
+                self.PLAY_MEDAL_DIAMOND_CLEARED: self.GAME_PLAY_MEDAL_DIAMOND_CLEARED,
+                self.PLAY_MEDAL_STAR_CLEARED: self.GAME_PLAY_MEDAL_STAR_CLEARED,
+                self.PLAY_MEDAL_CIRCLE_FULL_COMBO: self.GAME_PLAY_MEDAL_CIRCLE_FULL_COMBO,
+                self.PLAY_MEDAL_DIAMOND_FULL_COMBO: self.GAME_PLAY_MEDAL_DIAMOND_FULL_COMBO,
+                self.PLAY_MEDAL_STAR_FULL_COMBO: self.GAME_PLAY_MEDAL_STAR_FULL_COMBO,
+                self.PLAY_MEDAL_PERFECT: self.GAME_PLAY_MEDAL_PERFECT,
+            }[medal]))
+            music.add_child(Node.s32('old_score', 0))
+            music.add_child(Node.u8('old_clear_type', 0))
+
+        # Net VS section
+        netvs = Node.void('netvs')
+        root.add_child(netvs)
+        netvs.add_child(Node.s32('rank_point', 0))
+        netvs.add_child(Node.s16_array('record', [0, 0, 0, 0, 0, 0]))
+        netvs.add_child(Node.u8('rank', 0))
+        netvs.add_child(Node.s8('vs_rank_old', 0))
+        netvs.add_child(Node.s8_array('ojama_condition', [0] * 74))
+        netvs.add_child(Node.s8_array('set_ojama', [0, 0, 0]))
+        netvs.add_child(Node.s8_array('set_recommend', [0, 0, 0]))
+        netvs.add_child(Node.u32('netvs_play_cnt', 0))
+        for dialog in [0, 1, 2, 3, 4, 5]:
+            # TODO: Configure this, maybe?
+            netvs.add_child(Node.string('dialog', 'dialog#{}'.format(dialog)))
+
+        # Set up config
+        config = Node.void('config')
+        root.add_child(config)
+        config.add_child(Node.u8('mode', profile.get_int('mode', 0)))
+        config.add_child(Node.s16('chara', profile.get_int('chara', -1)))
+        config.add_child(Node.s16('music', profile.get_int('music', -1)))
+        config.add_child(Node.u8('sheet', profile.get_int('sheet', 0)))
+        config.add_child(Node.s8('category', profile.get_int('category', 1)))
+        config.add_child(Node.s8('sub_category', profile.get_int('sub_category', -1)))
+        config.add_child(Node.s8('chara_category', profile.get_int('chara_category', -1)))
+        config.add_child(Node.s16('story_id', profile.get_int('story_id', -1)))
+        config.add_child(Node.s16('course_id', profile.get_int('course_id', -1)))
+        config.add_child(Node.s8('course_folder', profile.get_int('course_folder', -1)))
+        config.add_child(Node.s8('story_folder', profile.get_int('story_folder', -1)))
+        config.add_child(Node.s8('ms_banner_disp', profile.get_int('ms_banner_disp')))
+        config.add_child(Node.s8('ms_down_info', profile.get_int('ms_down_info')))
+        config.add_child(Node.s8('ms_side_info', profile.get_int('ms_side_info')))
+        config.add_child(Node.s8('ms_raise_type', profile.get_int('ms_raise_type')))
+        config.add_child(Node.s8('ms_rnd_type', profile.get_int('ms_rnd_type')))
+
+        # Set up option
+        option_dict = profile.get_dict('option')
+        option = Node.void('option')
+        root.add_child(option)
+        option.add_child(Node.s16('hispeed', option_dict.get_int('hispeed', 10)))
+        option.add_child(Node.u8('popkun', option_dict.get_int('popkun', 0)))
+        option.add_child(Node.bool('hidden', option_dict.get_bool('hidden', False)))
+        option.add_child(Node.s16('hidden_rate', option_dict.get_int('hidden_rate', -1)))
+        option.add_child(Node.bool('sudden', option_dict.get_bool('sudden', False)))
+        option.add_child(Node.s16('sudden_rate', option_dict.get_int('sudden_rate', -1)))
+        option.add_child(Node.s8('randmir', option_dict.get_int('randmir', 0)))
+        option.add_child(Node.s8('gauge_type', option_dict.get_int('gauge_type', 0)))
+        option.add_child(Node.u8('ojama_0', option_dict.get_int('ojama_0', 0)))
+        option.add_child(Node.u8('ojama_1', option_dict.get_int('ojama_1', 0)))
+        option.add_child(Node.bool('forever_0', option_dict.get_bool('forever_0', False)))
+        option.add_child(Node.bool('forever_1', option_dict.get_bool('forever_1', False)))
+        option.add_child(Node.bool('full_setting', option_dict.get_bool('full_setting', False)))
+
+        # Set up info
+        info = Node.void('info')
+        root.add_child(info)
+        info.add_child(Node.u16('ep', profile.get_int('ep', 0)))
+        info.add_child(Node.u16('ap', profile.get_int('ap', 0)))
+
+        # Set up custom_cate
+        custom_cate = Node.void('custom_cate')
+        root.add_child(custom_cate)
+        custom_cate.add_child(Node.s8('valid', 0))
+        custom_cate.add_child(Node.s8('lv_min', -1))
+        custom_cate.add_child(Node.s8('lv_max', -1))
+        custom_cate.add_child(Node.s8('medal_min', -1))
+        custom_cate.add_child(Node.s8('medal_max', -1))
+        custom_cate.add_child(Node.s8('friend_no', -1))
+        custom_cate.add_child(Node.s8('score_flg', -1))
+
+        # Set up customize
+        customize_dict = profile.get_dict('customize')
+        customize = Node.void('customize')
+        root.add_child(customize)
+        customize.add_child(Node.u16('effect', customize_dict.get_int('effect')))
+        customize.add_child(Node.u16('hukidashi', customize_dict.get_int('hukidashi')))
+        customize.add_child(Node.u16('font', customize_dict.get_int('font')))
+        customize.add_child(Node.u16('comment_1', customize_dict.get_int('comment_1')))
+        customize.add_child(Node.u16('comment_2', customize_dict.get_int('comment_2')))
+
+        # Set up achievements
+        achievements = self.data.local.user.get_achievements(self.game, self.version, userid)
+        for achievement in achievements:
+            if achievement.type == 'item':
+                itemtype = achievement.data.get_int('type')
+                param = achievement.data.get_int('param')
+
+                item = Node.void('item')
+                root.add_child(item)
+                item.add_child(Node.u8('type', itemtype))
+                item.add_child(Node.u16('id', achievement.id))
+                item.add_child(Node.u16('param', param))
+                item.add_child(Node.bool('is_new', False))
+
+            elif achievement.type == 'achievement':
+                count = achievement.data.get_int('count')
+
+                ach_node = Node.void('achievement')
+                root.add_child(ach_node)
+                ach_node.add_child(Node.u8('type', achievement.id))
+                ach_node.add_child(Node.u32('count', count))
+
+            elif achievement.type == 'chara':
+                friendship = achievement.data.get_int('friendship')
+
+                chara = Node.void('chara_param')
+                root.add_child(chara)
+                chara.add_child(Node.u16('chara_id', achievement.id))
+                chara.add_child(Node.u16('friendship', friendship))
+
+            elif achievement.type == 'story':
+                chapter = achievement.data.get_int('chapter')
+                gauge = achievement.data.get_int('gauge')
+                cleared = achievement.data.get_bool('cleared')
+                clear_chapter = achievement.data.get_int('clear_chapter')
+
+                story = Node.void('story')
+                root.add_child(story)
+                story.add_child(Node.u32('story_id', achievement.id))
+                story.add_child(Node.u32('chapter_id', chapter))
+                story.add_child(Node.u16('gauge_point', gauge))
+                story.add_child(Node.bool('is_cleared', cleared))
+                story.add_child(Node.u32('clear_chapter', clear_chapter))
+
+        return root
+
+    def unformat_profile(self, userid: UserID, request: Node, oldprofile: ValidatedDict) -> ValidatedDict:
+        newprofile = copy.deepcopy(oldprofile)
+
+        account = request.child('account')
+        if account is not None:
+            newprofile.replace_int('tutorial', account.child_value('tutorial'))
+            newprofile.replace_int('read_news', account.child_value('read_news'))
+
+        info = request.child('info')
+        if info is not None:
+            newprofile.replace_int('ep', info.child_value('ep'))
+            newprofile.replace_int('ap', info.child_value('ap'))
+
+        config = request.child('config')
+        if config is not None:
+            newprofile.replace_int('mode', config.child_value('mode'))
+            newprofile.replace_int('chara', config.child_value('chara'))
+            newprofile.replace_int('music', config.child_value('music'))
+            newprofile.replace_int('sheet', config.child_value('sheet'))
+            newprofile.replace_int('category', config.child_value('category'))
+            newprofile.replace_int('sub_category', config.child_value('sub_category'))
+            newprofile.replace_int('chara_category', config.child_value('chara_category'))
+            newprofile.replace_int('story_id', config.child_value('story_id'))
+            newprofile.replace_int('course_id', config.child_value('course_id'))
+            newprofile.replace_int('course_folder', config.child_value('course_folder'))
+            newprofile.replace_int('story_folder', config.child_value('story_folder'))
+            newprofile.replace_int('ms_banner_disp', config.child_value('ms_banner_disp'))
+            newprofile.replace_int('ms_down_info', config.child_value('ms_down_info'))
+            newprofile.replace_int('ms_side_info', config.child_value('ms_side_info'))
+            newprofile.replace_int('ms_raise_type', config.child_value('ms_raise_type'))
+            newprofile.replace_int('ms_rnd_type', config.child_value('ms_rnd_type'))
+
+        option_dict = newprofile.get_dict('option')
+        option = request.child('option')
+        if option is not None:
+            option_dict.replace_int('hispeed', option.child_value('hispeed'))
+            option_dict.replace_int('popkun', option.child_value('popkun'))
+            option_dict.replace_bool('hidden', option.child_value('hidden'))
+            option_dict.replace_bool('sudden', option.child_value('sudden'))
+            option_dict.replace_int('hidden_rate', option.child_value('hidden_rate'))
+            option_dict.replace_int('sudden_rate', option.child_value('sudden_rate'))
+            option_dict.replace_int('randmir', option.child_value('randmir'))
+            option_dict.replace_int('gauge_type', option.child_value('gauge_type'))
+            option_dict.replace_int('ojama_0', option.child_value('ojama_0'))
+            option_dict.replace_int('ojama_1', option.child_value('ojama_1'))
+            option_dict.replace_bool('forever_0', option.child_value('forever_0'))
+            option_dict.replace_bool('forever_1', option.child_value('forever_1'))
+            option_dict.replace_bool('full_setting', option.child_value('full_setting'))
+        newprofile.replace_dict('option', option_dict)
+
+        customize_dict = newprofile.get_dict('customize')
+        customize = request.child('customize')
+        if customize is not None:
+            customize_dict.replace_int('effect', customize.child_value('effect'))
+            customize_dict.replace_int('hukidashi', customize.child_value('hukidashi'))
+            customize_dict.replace_int('font', customize.child_value('font'))
+            customize_dict.replace_int('comment_1', customize.child_value('comment_1'))
+            customize_dict.replace_int('comment_2', customize.child_value('comment_2'))
+        newprofile.replace_dict('customize', customize_dict)
+
+        # Keep track of play statistics
+        self.update_play_statistics(userid)
+
+        # Extract achievements
+        for node in request.children:
+            if node.name == 'item':
+                if not node.child_value('is_new'):
+                    # No need to save this one
+                    continue
+
+                itemid = node.child_value('id')
+                itemtype = node.child_value('type')
+                param = node.child_value('param')
+
+                self.data.local.user.put_achievement(
+                    self.game,
+                    self.version,
+                    userid,
+                    itemid,
+                    'item',
+                    {
+                        'type': itemtype,
+                        'param': param,
+                    },
+                )
+
+            elif node.name == 'achievement':
+                achievementid = node.child_value('type')
+                count = node.child_value('count')
+
+                self.data.local.user.put_achievement(
+                    self.game,
+                    self.version,
+                    userid,
+                    achievementid,
+                    'achievement',
+                    {
+                        'count': count,
+                    },
+                )
+
+            elif node.name == 'chara_param':
+                charaid = node.child_value('chara_id')
+                friendship = node.child_value('friendship')
+
+                self.data.local.user.put_achievement(
+                    self.game,
+                    self.version,
+                    userid,
+                    charaid,
+                    'chara',
+                    {
+                        'friendship': friendship,
+                    },
+                )
+
+            elif node.name == 'story':
+                storyid = node.child_value('story_id')
+                chapter = node.child_value('chapter_id')
+                gauge = node.child_value('gauge_point')
+                cleared = node.child_value('is_cleared')
+                clear_chapter = node.child_value('clear_chapter')
+
+                self.data.local.user.put_achievement(
+                    self.game,
+                    self.version,
+                    userid,
+                    storyid,
+                    'story',
+                    {
+                        'chapter': chapter,
+                        'gauge': gauge,
+                        'cleared': cleared,
+                        'clear_chapter': clear_chapter,
+                    },
+                )
+
+        return newprofile
+
+    def format_conversion(self, userid: UserID, profile: ValidatedDict) -> Node:
+        # Circular import, ugh
+        from bemani.backend.popn.eclale import PopnMusicEclale
+
+        root = Node.void('player23')
+        root.add_child(Node.string('name', profile.get_str('name', 'なし')))
+        root.add_child(Node.s16('chara', profile.get_int('chara', -1)))
+        root.add_child(Node.s8('result', 1))
+
+        scores = self.data.remote.music.get_scores(self.game, self.version, userid)
+        for score in scores:
+            if score.id > self.GAME_MAX_MUSIC_ID:
+                continue
+
+            # Skip any scores for chart types we don't support
+            if score.chart not in [
+                self.CHART_TYPE_EASY,
+                self.CHART_TYPE_NORMAL,
+                self.CHART_TYPE_HYPER,
+                self.CHART_TYPE_EX,
+            ]:
+                continue
+
+            points = score.points
+            medal = score.data.get_int('medal')
+
+            music = Node.void('music')
+            root.add_child(music)
+            music.add_child(Node.s16('music_num', score.id))
+            music.add_child(Node.u8('sheet_num', {
+                self.CHART_TYPE_EASY: PopnMusicEclale.GAME_CHART_TYPE_EASY,
+                self.CHART_TYPE_NORMAL: PopnMusicEclale.GAME_CHART_TYPE_NORMAL,
+                self.CHART_TYPE_HYPER: PopnMusicEclale.GAME_CHART_TYPE_HYPER,
+                self.CHART_TYPE_EX: PopnMusicEclale.GAME_CHART_TYPE_EX,
+            }[score.chart]))
+            music.add_child(Node.s32('score', points))
+            music.add_child(Node.u8('clear_type', {
+                self.PLAY_MEDAL_CIRCLE_FAILED: PopnMusicEclale.GAME_PLAY_MEDAL_CIRCLE_FAILED,
+                self.PLAY_MEDAL_DIAMOND_FAILED: PopnMusicEclale.GAME_PLAY_MEDAL_DIAMOND_FAILED,
+                self.PLAY_MEDAL_STAR_FAILED: PopnMusicEclale.GAME_PLAY_MEDAL_STAR_FAILED,
+                self.PLAY_MEDAL_EASY_CLEAR: PopnMusicEclale.GAME_PLAY_MEDAL_EASY_CLEAR,
+                self.PLAY_MEDAL_CIRCLE_CLEARED: PopnMusicEclale.GAME_PLAY_MEDAL_CIRCLE_CLEARED,
+                self.PLAY_MEDAL_DIAMOND_CLEARED: PopnMusicEclale.GAME_PLAY_MEDAL_DIAMOND_CLEARED,
+                self.PLAY_MEDAL_STAR_CLEARED: PopnMusicEclale.GAME_PLAY_MEDAL_STAR_CLEARED,
+                self.PLAY_MEDAL_CIRCLE_FULL_COMBO: PopnMusicEclale.GAME_PLAY_MEDAL_CIRCLE_FULL_COMBO,
+                self.PLAY_MEDAL_DIAMOND_FULL_COMBO: PopnMusicEclale.GAME_PLAY_MEDAL_DIAMOND_FULL_COMBO,
+                self.PLAY_MEDAL_STAR_FULL_COMBO: PopnMusicEclale.GAME_PLAY_MEDAL_STAR_FULL_COMBO,
+                self.PLAY_MEDAL_PERFECT: PopnMusicEclale.GAME_PLAY_MEDAL_PERFECT,
+            }[medal]))
+            music.add_child(Node.s16('cnt', score.plays))
+
+        return root

bemani/backend/popn/peace.py

@@ -0,0 +1,15 @@
+# vim: set fileencoding=utf-8
+from typing import Optional
+
+from bemani.backend.popn.base import PopnMusicBase
+from bemani.backend.popn.usaneko import PopnMusicUsaNeko
+from bemani.common import VersionConstants
+
+
+class PopnMusicPeace(PopnMusicBase):
+
+    name = "Pop'n Music peace"
+    version = VersionConstants.POPN_MUSIC_PEACE
+
+    def previous_version(self) -> Optional[PopnMusicBase]:
+        return PopnMusicUsaNeko(self.data, self.config, self.model)

bemani/backend/popn/stubs.py

@@ -0,0 +1,164 @@
+# vim: set fileencoding=utf-8
+from typing import Optional
+
+from bemani.backend.popn.base import PopnMusicBase
+from bemani.common import VersionConstants
+
+
+class PopnMusic(PopnMusicBase):
+
+    name = "Pop'n Music"
+    version = VersionConstants.POPN_MUSIC
+
+
+class PopnMusic2(PopnMusicBase):
+
+    name = "Pop'n Music 2"
+    version = VersionConstants.POPN_MUSIC_2
+
+    def previous_version(self) -> Optional[PopnMusicBase]:
+        return PopnMusic(self.data, self.config, self.model)
+
+
+class PopnMusic3(PopnMusicBase):
+
+    name = "Pop'n Music 3"
+    version = VersionConstants.POPN_MUSIC_3
+
+    def previous_version(self) -> Optional[PopnMusicBase]:
+        return PopnMusic2(self.data, self.config, self.model)
+
+
+class PopnMusic4(PopnMusicBase):
+
+    name = "Pop'n Music 4"
+    version = VersionConstants.POPN_MUSIC_4
+
+    def previous_version(self) -> Optional[PopnMusicBase]:
+        return PopnMusic3(self.data, self.config, self.model)
+
+
+class PopnMusic5(PopnMusicBase):
+
+    name = "Pop'n Music 5"
+    version = VersionConstants.POPN_MUSIC_5
+
+    def previous_version(self) -> Optional[PopnMusicBase]:
+        return PopnMusic4(self.data, self.config, self.model)
+
+
+class PopnMusic6(PopnMusicBase):
+
+    name = "Pop'n Music 6"
+    version = VersionConstants.POPN_MUSIC_6
+
+    def previous_version(self) -> Optional[PopnMusicBase]:
+        return PopnMusic5(self.data, self.config, self.model)
+
+
+class PopnMusic7(PopnMusicBase):
+
+    name = "Pop'n Music 7"
+    version = VersionConstants.POPN_MUSIC_7
+
+    def previous_version(self) -> Optional[PopnMusicBase]:
+        return PopnMusic6(self.data, self.config, self.model)
+
+
+class PopnMusic8(PopnMusicBase):
+
+    name = "Pop'n Music 8"
+    version = VersionConstants.POPN_MUSIC_8
+
+    def previous_version(self) -> Optional[PopnMusicBase]:
+        return PopnMusic7(self.data, self.config, self.model)
+
+
+class PopnMusic9(PopnMusicBase):
+
+    name = "Pop'n Music 9"
+    version = VersionConstants.POPN_MUSIC_9
+
+    def previous_version(self) -> Optional[PopnMusicBase]:
+        return PopnMusic8(self.data, self.config, self.model)
+
+
+class PopnMusic10(PopnMusicBase):
+
+    name = "Pop'n Music 10"
+    version = VersionConstants.POPN_MUSIC_10
+
+    def previous_version(self) -> Optional[PopnMusicBase]:
+        return PopnMusic9(self.data, self.config, self.model)
+
+
+class PopnMusic11(PopnMusicBase):
+
+    name = "Pop'n Music 11"
+    version = VersionConstants.POPN_MUSIC_11
+
+    def previous_version(self) -> Optional[PopnMusicBase]:
+        return PopnMusic10(self.data, self.config, self.model)
+
+
+class PopnMusicIroha(PopnMusicBase):
+
+    name = "Pop'n Music いろは"
+    version = VersionConstants.POPN_MUSIC_IROHA
+
+    def previous_version(self) -> Optional[PopnMusicBase]:
+        return PopnMusic11(self.data, self.config, self.model)
+
+
+class PopnMusicCarnival(PopnMusicBase):
+
+    name = "Pop'n Music カーニバル"
+    version = VersionConstants.POPN_MUSIC_CARNIVAL
+
+    def previous_version(self) -> Optional[PopnMusicBase]:
+        return PopnMusicIroha(self.data, self.config, self.model)
+
+
+class PopnMusicFever(PopnMusicBase):
+
+    name = "Pop'n Music FEVER!"
+    version = VersionConstants.POPN_MUSIC_FEVER
+
+    def previous_version(self) -> Optional[PopnMusicBase]:
+        return PopnMusicCarnival(self.data, self.config, self.model)
+
+
+class PopnMusicAdventure(PopnMusicBase):
+
+    name = "Pop'n Music ADVENTURE"
+    version = VersionConstants.POPN_MUSIC_ADVENTURE
+
+    def previous_version(self) -> Optional[PopnMusicBase]:
+        return PopnMusicFever(self.data, self.config, self.model)
+
+
+class PopnMusicParty(PopnMusicBase):
+
+    name = "Pop'n Music Party♪"
+    version = VersionConstants.POPN_MUSIC_PARTY
+
+    def previous_version(self) -> Optional[PopnMusicBase]:
+        return PopnMusicAdventure(self.data, self.config, self.model)
+
+
+class PopnMusicTheMovie(PopnMusicBase):
+
+    name = "Pop'n Music THE MOVIE"
+    version = VersionConstants.POPN_MUSIC_THE_MOVIE
+
+    def previous_version(self) -> Optional[PopnMusicBase]:
+        return PopnMusicParty(self.data, self.config, self.model)
+
+
+class PopnMusicSengokuRetsuden(PopnMusicBase):
+
+    name = "Pop'n Music せんごく列伝"
+    version = VersionConstants.POPN_MUSIC_SENGOKU_RETSUDEN
+
+    def previous_version(self) -> Optional[PopnMusicBase]:
+        return PopnMusicTheMovie(self.data, self.config, self.model)

bemani/backend/popn/sunnypark.py

@@ -0,0 +1,529 @@
+# vim: set fileencoding=utf-8
+import copy
+from typing import Optional
+
+from bemani.backend.popn.base import PopnMusicBase
+from bemani.backend.popn.fantasia import PopnMusicFantasia
+
+from bemani.backend.base import Status
+from bemani.common import ValidatedDict, VersionConstants, Time, ID
+from bemani.data import UserID
+from bemani.protocol import Node
+
+
+class PopnMusicSunnyPark(PopnMusicBase):
+
+    name = "Pop'n Music Sunny Park"
+    version = VersionConstants.POPN_MUSIC_SUNNY_PARK
+
+    # Chart type, as returned from the game
+    GAME_CHART_TYPE_EASY = 0
+    GAME_CHART_TYPE_NORMAL = 1
+    GAME_CHART_TYPE_HYPER = 2
+    GAME_CHART_TYPE_EX = 3
+
+    # Chart type, as packed into a hiscore binary
+    GAME_CHART_TYPE_EASY_POSITION = 0
+    GAME_CHART_TYPE_NORMAL_POSITION = 1
+    GAME_CHART_TYPE_HYPER_POSITION = 2
+    GAME_CHART_TYPE_EX_POSITION = 3
+
+    # Medal type, as returned from the game
+    GAME_PLAY_MEDAL_CIRCLE_FAILED = 1
+    GAME_PLAY_MEDAL_DIAMOND_FAILED = 2
+    GAME_PLAY_MEDAL_STAR_FAILED = 3
+    GAME_PLAY_MEDAL_CIRCLE_CLEARED = 5
+    GAME_PLAY_MEDAL_DIAMOND_CLEARED = 6
+    GAME_PLAY_MEDAL_STAR_CLEARED = 7
+    GAME_PLAY_MEDAL_CIRCLE_FULL_COMBO = 9
+    GAME_PLAY_MEDAL_DIAMOND_FULL_COMBO = 10
+    GAME_PLAY_MEDAL_STAR_FULL_COMBO = 11
+    GAME_PLAY_MEDAL_PERFECT = 15
+
+    # Maximum music ID for this game
+    GAME_MAX_MUSIC_ID = 1350
+
+    def previous_version(self) -> Optional[PopnMusicBase]:
+        return PopnMusicFantasia(self.data, self.config, self.model)
+
+    def format_profile(self, userid: UserID, profile: ValidatedDict) -> Node:
+        root = Node.void('playerdata')
+
+        # Set up the base profile
+        base = Node.void('base')
+        root.add_child(base)
+        base.add_child(Node.string('name', profile.get_str('name', 'なし')))
+        base.add_child(Node.string('g_pm_id', ID.format_extid(profile.get_int('extid'))))
+        base.add_child(Node.u8('mode', profile.get_int('mode', 0)))
+        base.add_child(Node.s8('button', profile.get_int('button', 0)))
+        base.add_child(Node.s8('last_play_flag', profile.get_int('last_play_flag', -1)))
+        base.add_child(Node.u8('medal_and_friend', profile.get_int('medal_and_friend', 0)))
+        base.add_child(Node.s8('category', profile.get_int('category', -1)))
+        base.add_child(Node.s8('sub_category', profile.get_int('sub_category', -1)))
+        base.add_child(Node.s16('chara', profile.get_int('chara', -1)))
+        base.add_child(Node.s8('chara_category', profile.get_int('chara_category', -1)))
+        base.add_child(Node.u8('collabo', 255))
+        base.add_child(Node.u8('sheet', profile.get_int('sheet', 0)))
+        base.add_child(Node.s8('tutorial', profile.get_int('tutorial', 0)))
+        base.add_child(Node.s8('music_open_pt', profile.get_int('music_open_pt', 0)))
+        base.add_child(Node.s8('is_conv', -1))
+        base.add_child(Node.s32('option', profile.get_int('option', 0)))
+        base.add_child(Node.s16('music', profile.get_int('music', -1)))
+        base.add_child(Node.u16('ep', profile.get_int('ep', 0)))
+        base.add_child(Node.s32_array('sp_color_flg', profile.get_int_array('sp_color_flg', 2)))
+        base.add_child(Node.s32('read_news', profile.get_int('read_news', 0)))
+        base.add_child(Node.s16('consecutive_days_coupon', profile.get_int('consecutive_days_coupon', 0)))
+        base.add_child(Node.s8('staff', 0))
+        # These are probably from an old event, but if they aren't present and defaulted,
+        # then different songs show up in the Zoo event.
+        base.add_child(Node.u16_array('gitadora_point', profile.get_int_array('gitadora_point', 3, [2000, 2000, 2000])))
+        base.add_child(Node.u8('gitadora_select', profile.get_int('gitadora_select', 2)))
+
+        # Statistics section and scores section
+        statistics = self.get_play_statistics(userid)
+        last_play_date = statistics.get_int_array('last_play_date', 3)
+        today_play_date = Time.todays_date()
+        if (
+            last_play_date[0] == today_play_date[0] and
+            last_play_date[1] == today_play_date[1] and
+            last_play_date[2] == today_play_date[2]
+        ):
+            today_count = statistics.get_int('today_plays', 0)
+        else:
+            today_count = 0
+        base.add_child(Node.u8('active_fr_num', 0))  # TODO: Hook up rivals code?
+        base.add_child(Node.s32('total_play_cnt', statistics.get_int('total_plays', 0)))
+        base.add_child(Node.s16('today_play_cnt', today_count))
+        base.add_child(Node.s16('consecutive_days', statistics.get_int('consecutive_days', 0)))
+
+        last_played = [x[0] for x in self.data.local.music.get_last_played(self.game, self.version, userid, 3)]
+        most_played = [x[0] for x in self.data.local.music.get_most_played(self.game, self.version, userid, 20)]
+        while len(last_played) < 3:
+            last_played.append(-1)
+        while len(most_played) < 20:
+            most_played.append(-1)
+
+        hiscore_array = [0] * int((((self.GAME_MAX_MUSIC_ID * 4) * 17) + 7) / 8)
+        clear_medal = [0] * self.GAME_MAX_MUSIC_ID
+        clear_medal_sub = [0] * self.GAME_MAX_MUSIC_ID
+
+        scores = self.data.remote.music.get_scores(self.game, self.version, userid)
+        for score in scores:
+            if score.id > self.GAME_MAX_MUSIC_ID:
+                continue
+
+            # Skip any scores for chart types we don't support
+            if score.chart not in [
+                self.CHART_TYPE_EASY,
+                self.CHART_TYPE_NORMAL,
+                self.CHART_TYPE_HYPER,
+                self.CHART_TYPE_EX,
+            ]:
+                continue
+
+            points = score.points
+            medal = {
+                self.PLAY_MEDAL_CIRCLE_FAILED: self.GAME_PLAY_MEDAL_CIRCLE_FAILED,
+                self.PLAY_MEDAL_DIAMOND_FAILED: self.GAME_PLAY_MEDAL_DIAMOND_FAILED,
+                self.PLAY_MEDAL_STAR_FAILED: self.GAME_PLAY_MEDAL_STAR_FAILED,
+                self.PLAY_MEDAL_EASY_CLEAR: self.GAME_PLAY_MEDAL_CIRCLE_CLEARED,  # Map approximately
+                self.PLAY_MEDAL_CIRCLE_CLEARED: self.GAME_PLAY_MEDAL_CIRCLE_CLEARED,
+                self.PLAY_MEDAL_DIAMOND_CLEARED: self.GAME_PLAY_MEDAL_DIAMOND_CLEARED,
+                self.PLAY_MEDAL_STAR_CLEARED: self.GAME_PLAY_MEDAL_STAR_CLEARED,
+                self.PLAY_MEDAL_CIRCLE_FULL_COMBO: self.GAME_PLAY_MEDAL_CIRCLE_FULL_COMBO,
+                self.PLAY_MEDAL_DIAMOND_FULL_COMBO: self.GAME_PLAY_MEDAL_DIAMOND_FULL_COMBO,
+                self.PLAY_MEDAL_STAR_FULL_COMBO: self.GAME_PLAY_MEDAL_STAR_FULL_COMBO,
+                self.PLAY_MEDAL_PERFECT: self.GAME_PLAY_MEDAL_PERFECT,
+            }[score.data.get_int('medal')]
+            clear_medal[score.id] = clear_medal[score.id] | (medal << (score.chart * 4))
+
+            hiscore_index = (score.id * 4) + {
+                self.CHART_TYPE_EASY: self.GAME_CHART_TYPE_EASY_POSITION,
+                self.CHART_TYPE_NORMAL: self.GAME_CHART_TYPE_NORMAL_POSITION,
+                self.CHART_TYPE_HYPER: self.GAME_CHART_TYPE_HYPER_POSITION,
+                self.CHART_TYPE_EX: self.GAME_CHART_TYPE_EX_POSITION,
+            }[score.chart]
+            hiscore_byte_pos = int((hiscore_index * 17) / 8)
+            hiscore_bit_pos = int((hiscore_index * 17) % 8)
+            hiscore_value = points << hiscore_bit_pos
+            hiscore_array[hiscore_byte_pos] = hiscore_array[hiscore_byte_pos] | (hiscore_value & 0xFF)
+            hiscore_array[hiscore_byte_pos + 1] = hiscore_array[hiscore_byte_pos + 1] | ((hiscore_value >> 8) & 0xFF)
+            hiscore_array[hiscore_byte_pos + 2] = hiscore_array[hiscore_byte_pos + 2] | ((hiscore_value >> 16) & 0xFF)
+
+        hiscore = bytes(hiscore_array)
+
+        base.add_child(Node.s16_array('my_best', most_played))
+        base.add_child(Node.s16_array('latest_music', last_played))
+        base.add_child(Node.u16_array('clear_medal', clear_medal))
+        base.add_child(Node.u8_array('clear_medal_sub', clear_medal_sub))
+
+        # Goes outside of base for some reason
+        root.add_child(Node.binary('hiscore', hiscore))
+
+        # Avatar section
+        avatar_dict = profile.get_dict('avatar')
+        avatar = Node.void('avatar')
+        root.add_child(avatar)
+        avatar.add_child(Node.u8('hair', avatar_dict.get_int('hair', 0)))
+        avatar.add_child(Node.u8('face', avatar_dict.get_int('face', 0)))
+        avatar.add_child(Node.u8('body', avatar_dict.get_int('body', 0)))
+        avatar.add_child(Node.u8('effect', avatar_dict.get_int('effect', 0)))
+        avatar.add_child(Node.u8('object', avatar_dict.get_int('object', 0)))
+        avatar.add_child(Node.u8_array('comment', avatar_dict.get_int_array('comment', 2)))
+        avatar.add_child(Node.s32_array('get_hair', avatar_dict.get_int_array('get_hair', 2)))
+        avatar.add_child(Node.s32_array('get_face', avatar_dict.get_int_array('get_face', 2)))
+        avatar.add_child(Node.s32_array('get_body', avatar_dict.get_int_array('get_body', 2)))
+        avatar.add_child(Node.s32_array('get_effect', avatar_dict.get_int_array('get_effect', 2)))
+        avatar.add_child(Node.s32_array('get_object', avatar_dict.get_int_array('get_object', 2)))
+        avatar.add_child(Node.s32_array('get_comment_over', avatar_dict.get_int_array('get_comment_over', 3)))
+        avatar.add_child(Node.s32_array('get_comment_under', avatar_dict.get_int_array('get_comment_under', 3)))
+
+        # Avatar add section
+        avatar_add_dict = profile.get_dict('avatar_add')
+        avatar_add = Node.void('avatar_add')
+        root.add_child(avatar_add)
+        avatar_add.add_child(Node.s32_array('get_hair', avatar_add_dict.get_int_array('get_hair', 2)))
+        avatar_add.add_child(Node.s32_array('get_face', avatar_add_dict.get_int_array('get_face', 2)))
+        avatar_add.add_child(Node.s32_array('get_body', avatar_add_dict.get_int_array('get_body', 2)))
+        avatar_add.add_child(Node.s32_array('get_effect', avatar_add_dict.get_int_array('get_effect', 2)))
+        avatar_add.add_child(Node.s32_array('get_object', avatar_add_dict.get_int_array('get_object', 2)))
+        avatar_add.add_child(Node.s32_array('get_comment_over', avatar_add_dict.get_int_array('get_comment_over', 2)))
+        avatar_add.add_child(Node.s32_array('get_comment_under', avatar_add_dict.get_int_array('get_comment_under', 2)))
+        avatar_add.add_child(Node.s32_array('new_hair', avatar_add_dict.get_int_array('new_hair', 2)))
+        avatar_add.add_child(Node.s32_array('new_face', avatar_add_dict.get_int_array('new_face', 2)))
+        avatar_add.add_child(Node.s32_array('new_body', avatar_add_dict.get_int_array('new_body', 2)))
+        avatar_add.add_child(Node.s32_array('new_effect', avatar_add_dict.get_int_array('new_effect', 2)))
+        avatar_add.add_child(Node.s32_array('new_object', avatar_add_dict.get_int_array('new_object', 2)))
+        avatar_add.add_child(Node.s32_array('new_comment_over', avatar_add_dict.get_int_array('new_comment_over', 2)))
+        avatar_add.add_child(Node.s32_array('new_comment_under', avatar_add_dict.get_int_array('new_comment_under', 2)))
+
+        # Net VS section
+        netvs = Node.void('netvs')
+        root.add_child(netvs)
+        netvs.add_child(Node.s32('rank_point', 0))
+        netvs.add_child(Node.s16_array('record', [0, 0, 0, 0, 0, 0]))
+        netvs.add_child(Node.u8('rank', 0))
+        netvs.add_child(Node.s8('vs_rank_old', 0))
+        netvs.add_child(Node.s8_array('ojama_condition', [0] * 74))
+        netvs.add_child(Node.s8_array('set_ojama', [0, 0, 0]))
+        netvs.add_child(Node.s8_array('set_recommend', [0, 0, 0]))
+        netvs.add_child(Node.u8('netvs_play_cnt', 0))
+        for dialog in [0, 1, 2, 3, 4, 5]:
+            # TODO: Configure this, maybe?
+            netvs.add_child(Node.string('dialog', 'dialog#{}'.format(dialog)))
+
+        sp_data = Node.void('sp_data')
+        root.add_child(sp_data)
+        sp_data.add_child(Node.s32('sp', profile.get_int('sp', 0)))
+
+        gakuen = Node.void('gakuen_data')
+        root.add_child(gakuen)
+        gakuen.add_child(Node.s32('music_list', -1))
+
+        saucer = Node.void('flying_saucer')
+        root.add_child(saucer)
+        saucer.add_child(Node.s32('music_list', -1))
+        saucer.add_child(Node.s32('tune_count', -1))
+        saucer.add_child(Node.u32('clear_norma', 0))
+        saucer.add_child(Node.u32('clear_norma_add', 0))
+
+        zoo_dict = profile.get_dict('zoo')
+        zoo = Node.void('zoo')
+        root.add_child(zoo)
+        zoo.add_child(Node.u16_array('point', zoo_dict.get_int_array('point', 5)))
+        zoo.add_child(Node.s32_array('music_list', zoo_dict.get_int_array('music_list', 2)))
+        zoo.add_child(Node.s8_array('today_play_flag', zoo_dict.get_int_array('today_play_flag', 4)))
+
+        triple = Node.void('triple_journey')
+        root.add_child(triple)
+        triple.add_child(Node.s32('music_list', -1))
+        triple.add_child(Node.s32_array('boss_damage', [65534, 65534, 65534, 65534]))
+        triple.add_child(Node.s32_array('boss_stun', [0, 0, 0, 0]))
+        triple.add_child(Node.s32('magic_gauge', 0))
+        triple.add_child(Node.s32('today_party', 0))
+        triple.add_child(Node.bool('union_magic', False))
+        triple.add_child(Node.float('base_attack_rate', 1.0))
+        triple.add_child(Node.s32('iidx_play_num', 0))
+        triple.add_child(Node.s32('reflec_play_num', 0))
+        triple.add_child(Node.s32('voltex_play_num', 0))
+        triple.add_child(Node.bool('iidx_play_flg', True))
+        triple.add_child(Node.bool('reflec_play_flg', True))
+        triple.add_child(Node.bool('voltex_play_flg', True))
+
+        ios = Node.void('ios')
+        root.add_child(ios)
+        ios.add_child(Node.s32('continueRightAnswer', 30))
+        ios.add_child(Node.s32('totalRightAnswer', 30))
+
+        kac2013 = Node.void('kac2013')
+        root.add_child(kac2013)
+        kac2013.add_child(Node.s8('music_num', 0))
+        kac2013.add_child(Node.s16('music', 0))
+        kac2013.add_child(Node.u8('sheet', 0))
+
+        baseball = Node.void('baseball_data')
+        root.add_child(baseball)
+        baseball.add_child(Node.s64('music_list', -1))
+
+        for id in [3, 5, 7]:
+            node = Node.void('floor_infection')
+            root.add_child(node)
+            node.add_child(Node.s32('infection_id', id))
+            node.add_child(Node.s32('music_list', -1))
+
+        return root
+
+    def format_conversion(self, userid: UserID, profile: ValidatedDict) -> Node:
+        # Circular import, ugh
+        from bemani.backend.popn.lapistoria import PopnMusicLapistoria
+
+        root = Node.void('playerdata')
+
+        root.add_child(Node.string('name', profile.get_str('name', 'なし')))
+        root.add_child(Node.s16('chara', profile.get_int('chara', -1)))
+        root.add_child(Node.s32('option', profile.get_int('option', 0)))
+        root.add_child(Node.s8('result', 1))
+
+        scores = self.data.remote.music.get_scores(self.game, self.version, userid)
+        for score in scores:
+            if score.id > self.GAME_MAX_MUSIC_ID:
+                continue
+
+            # Skip any scores for chart types we don't support
+            if score.chart not in [
+                self.CHART_TYPE_EASY,
+                self.CHART_TYPE_NORMAL,
+                self.CHART_TYPE_HYPER,
+                self.CHART_TYPE_EX,
+            ]:
+                continue
+
+            points = score.points
+            medal = score.data.get_int('medal')
+
+            music = Node.void('music')
+            root.add_child(music)
+            music.add_child(Node.s16('music_num', score.id))
+            music.add_child(Node.u8('sheet_num', {
+                self.CHART_TYPE_EASY: PopnMusicLapistoria.GAME_CHART_TYPE_EASY,
+                self.CHART_TYPE_NORMAL: PopnMusicLapistoria.GAME_CHART_TYPE_NORMAL,
+                self.CHART_TYPE_HYPER: PopnMusicLapistoria.GAME_CHART_TYPE_HYPER,
+                self.CHART_TYPE_EX: PopnMusicLapistoria.GAME_CHART_TYPE_EX,
+            }[score.chart]))
+            music.add_child(Node.s16('cnt', score.plays))
+            music.add_child(Node.s32('score', 0))
+            music.add_child(Node.u8('clear_type', 0))
+            music.add_child(Node.s32('old_score', points))
+            music.add_child(Node.u8('old_clear_type', {
+                self.PLAY_MEDAL_CIRCLE_FAILED: PopnMusicLapistoria.GAME_PLAY_MEDAL_CIRCLE_FAILED,
+                self.PLAY_MEDAL_DIAMOND_FAILED: PopnMusicLapistoria.GAME_PLAY_MEDAL_DIAMOND_FAILED,
+                self.PLAY_MEDAL_STAR_FAILED: PopnMusicLapistoria.GAME_PLAY_MEDAL_STAR_FAILED,
+                self.PLAY_MEDAL_EASY_CLEAR: PopnMusicLapistoria.GAME_PLAY_MEDAL_EASY_CLEAR,
+                self.PLAY_MEDAL_CIRCLE_CLEARED: PopnMusicLapistoria.GAME_PLAY_MEDAL_CIRCLE_CLEARED,
+                self.PLAY_MEDAL_DIAMOND_CLEARED: PopnMusicLapistoria.GAME_PLAY_MEDAL_DIAMOND_CLEARED,
+                self.PLAY_MEDAL_STAR_CLEARED: PopnMusicLapistoria.GAME_PLAY_MEDAL_STAR_CLEARED,
+                self.PLAY_MEDAL_CIRCLE_FULL_COMBO: PopnMusicLapistoria.GAME_PLAY_MEDAL_CIRCLE_FULL_COMBO,
+                self.PLAY_MEDAL_DIAMOND_FULL_COMBO: PopnMusicLapistoria.GAME_PLAY_MEDAL_DIAMOND_FULL_COMBO,
+                self.PLAY_MEDAL_STAR_FULL_COMBO: PopnMusicLapistoria.GAME_PLAY_MEDAL_STAR_FULL_COMBO,
+                self.PLAY_MEDAL_PERFECT: PopnMusicLapistoria.GAME_PLAY_MEDAL_PERFECT,
+            }[medal]))
+
+        return root
+
+    def unformat_profile(self, userid: UserID, request: Node, oldprofile: ValidatedDict) -> ValidatedDict:
+        newprofile = copy.deepcopy(oldprofile)
+        newprofile.replace_int('option', request.child_value('option'))
+        newprofile.replace_int('chara', request.child_value('chara'))
+        newprofile.replace_int('mode', request.child_value('mode'))
+        newprofile.replace_int('button', request.child_value('button'))
+        newprofile.replace_int('music', request.child_value('music'))
+        newprofile.replace_int('sheet', request.child_value('sheet'))
+        newprofile.replace_int('last_play_flag', request.child_value('last_play_flag'))
+        newprofile.replace_int('category', request.child_value('category'))
+        newprofile.replace_int('sub_category', request.child_value('sub_category'))
+        newprofile.replace_int('chara_category', request.child_value('chara_category'))
+        newprofile.replace_int('medal_and_friend', request.child_value('medal_and_friend'))
+        newprofile.replace_int('ep', request.child_value('ep'))
+        newprofile.replace_int_array('sp_color_flg', 2, request.child_value('sp_color_flg'))
+        newprofile.replace_int('read_news', request.child_value('read_news'))
+        newprofile.replace_int('consecutive_days_coupon', request.child_value('consecutive_days_coupon'))
+        newprofile.replace_int('tutorial', request.child_value('tutorial'))
+        newprofile.replace_int('music_open_pt', request.child_value('music_open_pt'))
+        newprofile.replace_int_array('gitadora_point', 3, request.child_value('gitadora_point'))
+        newprofile.replace_int('gitadora_select', request.child_value('gitadora_select'))
+
+        sp_node = request.child('sp_data')
+        if sp_node is not None:
+            newprofile.replace_int('sp', sp_node.child_value('sp'))
+
+        zoo_dict = newprofile.get_dict('zoo')
+        zoo_node = request.child('zoo')
+        if zoo_node is not None:
+            zoo_dict.replace_int_array('point', 5, zoo_node.child_value('point'))
+            zoo_dict.replace_int_array('music_list', 2, zoo_node.child_value('music_list'))
+            zoo_dict.replace_int_array('today_play_flag', 4, zoo_node.child_value('today_play_flag'))
+        newprofile.replace_dict('zoo', zoo_dict)
+
+        avatar_dict = newprofile.get_dict('avatar')
+        avatar_dict.replace_int('hair', request.child_value('hair'))
+        avatar_dict.replace_int('face', request.child_value('face'))
+        avatar_dict.replace_int('body', request.child_value('body'))
+        avatar_dict.replace_int('effect', request.child_value('effect'))
+        avatar_dict.replace_int('object', request.child_value('object'))
+        avatar_dict.replace_int_array('comment', 2, request.child_value('comment'))
+        avatar_dict.replace_int_array('get_hair', 2, request.child_value('get_hair'))
+        avatar_dict.replace_int_array('get_face', 2, request.child_value('get_face'))
+        avatar_dict.replace_int_array('get_body', 2, request.child_value('get_body'))
+        avatar_dict.replace_int_array('get_effect', 2, request.child_value('get_effect'))
+        avatar_dict.replace_int_array('get_object', 2, request.child_value('get_object'))
+        avatar_dict.replace_int_array('get_comment_over', 3, request.child_value('get_comment_over'))
+        avatar_dict.replace_int_array('get_comment_under', 3, request.child_value('get_comment_under'))
+        newprofile.replace_dict('avatar', avatar_dict)
+
+        avatar_add_dict = newprofile.get_dict('avatar_add')
+        avatar_add_node = request.child('avatar_add')
+        if avatar_add_node is not None:
+            avatar_add_dict.replace_int_array('get_hair', 2, avatar_add_node.child_value('get_hair'))
+            avatar_add_dict.replace_int_array('get_face', 2, avatar_add_node.child_value('get_face'))
+            avatar_add_dict.replace_int_array('get_body', 2, avatar_add_node.child_value('get_body'))
+            avatar_add_dict.replace_int_array('get_effect', 2, avatar_add_node.child_value('get_effect'))
+            avatar_add_dict.replace_int_array('get_object', 2, avatar_add_node.child_value('get_object'))
+            avatar_add_dict.replace_int_array('get_comment_over', 2, avatar_add_node.child_value('get_comment_over'))
+            avatar_add_dict.replace_int_array('get_comment_under', 2, avatar_add_node.child_value('get_comment_under'))
+            avatar_add_dict.replace_int_array('new_hair', 2, avatar_add_node.child_value('new_hair'))
+            avatar_add_dict.replace_int_array('new_face', 2, avatar_add_node.child_value('new_face'))
+            avatar_add_dict.replace_int_array('new_body', 2, avatar_add_node.child_value('new_body'))
+            avatar_add_dict.replace_int_array('new_effect', 2, avatar_add_node.child_value('new_effect'))
+            avatar_add_dict.replace_int_array('new_object', 2, avatar_add_node.child_value('new_object'))
+            avatar_add_dict.replace_int_array('new_comment_over', 2, avatar_add_node.child_value('new_comment_over'))
+            avatar_add_dict.replace_int_array('new_comment_under', 2, avatar_add_node.child_value('new_comment_under'))
+        newprofile.replace_dict('avatar_add', avatar_add_dict)
+
+        # Keep track of play statistics
+        self.update_play_statistics(userid)
+
+        # Extract scores
+        for node in request.children:
+            if node.name == 'stage':
+                songid = node.child_value('no')
+                chart = {
+                    self.GAME_CHART_TYPE_EASY: self.CHART_TYPE_EASY,
+                    self.GAME_CHART_TYPE_NORMAL: self.CHART_TYPE_NORMAL,
+                    self.GAME_CHART_TYPE_HYPER: self.CHART_TYPE_HYPER,
+                    self.GAME_CHART_TYPE_EX: self.CHART_TYPE_EX,
+                }[node.child_value('sheet')]
+                medal = (node.child_value('n_data') >> (chart * 4)) & 0x000F
+                medal = {
+                    self.GAME_PLAY_MEDAL_CIRCLE_FAILED: self.PLAY_MEDAL_CIRCLE_FAILED,
+                    self.GAME_PLAY_MEDAL_DIAMOND_FAILED: self.PLAY_MEDAL_DIAMOND_FAILED,
+                    self.GAME_PLAY_MEDAL_STAR_FAILED: self.PLAY_MEDAL_STAR_FAILED,
+                    self.GAME_PLAY_MEDAL_CIRCLE_CLEARED: self.PLAY_MEDAL_CIRCLE_CLEARED,
+                    self.GAME_PLAY_MEDAL_DIAMOND_CLEARED: self.PLAY_MEDAL_DIAMOND_CLEARED,
+                    self.GAME_PLAY_MEDAL_STAR_CLEARED: self.PLAY_MEDAL_STAR_CLEARED,
+                    self.GAME_PLAY_MEDAL_CIRCLE_FULL_COMBO: self.PLAY_MEDAL_CIRCLE_FULL_COMBO,
+                    self.GAME_PLAY_MEDAL_DIAMOND_FULL_COMBO: self.PLAY_MEDAL_DIAMOND_FULL_COMBO,
+                    self.GAME_PLAY_MEDAL_STAR_FULL_COMBO: self.PLAY_MEDAL_STAR_FULL_COMBO,
+                    self.GAME_PLAY_MEDAL_PERFECT: self.PLAY_MEDAL_PERFECT,
+                }[medal]
+                points = node.child_value('score')
+                self.update_score(userid, songid, chart, points, medal)
+
+        return newprofile
+
+    def handle_game_request(self, request: Node) -> Optional[Node]:
+        method = request.attribute('method')
+
+        if method == 'get':
+            # TODO: Hook these up to config so we can change this
+            root = Node.void('game')
+            root.add_child(Node.s32('ir_phase', 0))
+            root.add_child(Node.s32('music_open_phase', 8))
+            root.add_child(Node.s32('collabo_phase', 8))
+            root.add_child(Node.s32('personal_event_phase', 10))
+            root.add_child(Node.s32('shop_event_phase', 6))
+            root.add_child(Node.s32('netvs_phase', 0))
+            root.add_child(Node.s32('card_phase', 9))
+            root.add_child(Node.s32('other_phase', 9))
+            root.add_child(Node.s32('local_matching_enable', 1))
+            root.add_child(Node.s32('n_matching_sec', 60))
+            root.add_child(Node.s32('l_matching_sec', 60))
+            root.add_child(Node.s32('is_check_cpu', 0))
+            root.add_child(Node.s32('week_no', 0))
+            root.add_child(Node.s16_array('sel_ranking', [-1, -1, -1, -1, -1]))
+            root.add_child(Node.s16_array('up_ranking', [-1, -1, -1, -1, -1]))
+            return root
+
+        if method == 'active':
+            # Update the name of this cab for admin purposes
+            self.update_machine_name(request.child_value('shop_name'))
+            return Node.void('game')
+
+        if method == 'taxphase':
+            return Node.void('game')
+
+        # Invalid method
+        return None
+
+    def handle_playerdata_request(self, request: Node) -> Optional[Node]:
+        method = request.attribute('method')
+
+        if method == 'expire':
+            return Node.void('playerdata')
+
+        elif method == 'logout':
+            return Node.void('playerdata')
+
+        elif method == 'get':
+            modelstring = request.attribute('model')
+            refid = request.child_value('ref_id')
+            root = self.get_profile_by_refid(
+                refid,
+                self.NEW_PROFILE_ONLY if modelstring is None else self.OLD_PROFILE_ONLY,
+            )
+            if root is None:
+                root = Node.void('playerdata')
+                root.set_attribute('status', str(Status.NO_PROFILE))
+            return root
+
+        elif method == 'conversion':
+            refid = request.child_value('ref_id')
+            name = request.child_value('name')
+            chara = request.child_value('chara')
+            root = self.new_profile_by_refid(refid, name, chara)
+            if root is None:
+                root = Node.void('playerdata')
+                root.set_attribute('status', str(Status.NO_PROFILE))
+            return root
+
+        elif method == 'new':
+            refid = request.child_value('ref_id')
+            name = request.child_value('name')
+            root = self.new_profile_by_refid(refid, name)
+            if root is None:
+                root = Node.void('playerdata')
+                root.set_attribute('status', str(Status.NO_PROFILE))
+            return root
+
+        elif method == 'set':
+            refid = request.attribute('ref_id')
+
+            root = Node.void('playerdata')
+            root.add_child(Node.s8('pref', -1))
+            if refid is None:
+                return root
+
+            userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+            if userid is None:
+                return root
+
+            oldprofile = self.get_profile(userid) or ValidatedDict()
+            newprofile = self.unformat_profile(userid, request, oldprofile)
+
+            if newprofile is not None:
+                self.put_profile(userid, newprofile)
+                root.add_child(Node.string('name', newprofile['name']))
+
+            return root
+
+        # Invalid method
+        return None

bemani/backend/popn/tunestreet.py

@@ -0,0 +1,401 @@
+# vim: set fileencoding=utf-8
+import copy
+from typing import Optional
+
+from bemani.backend.popn.base import PopnMusicBase
+from bemani.backend.popn.stubs import PopnMusicSengokuRetsuden
+
+from bemani.backend.base import Status
+from bemani.common import ValidatedDict, VersionConstants
+from bemani.data import Score, UserID
+from bemani.protocol import Node
+
+
+class PopnMusicTuneStreet(PopnMusicBase):
+
+    name = "Pop'n Music TUNE STREET"
+    version = VersionConstants.POPN_MUSIC_TUNE_STREET
+
+    # Play modes, as reported by profile save from the game
+    GAME_PLAY_MODE_CHALLENGE = 3
+    GAME_PLAY_MODE_CHO_CHALLENGE = 4
+
+    # Play flags, as saved into/loaded from the DB
+    GAME_PLAY_FLAG_FAILED = 0
+    GAME_PLAY_FLAG_CLEARED = 1
+    GAME_PLAY_FLAG_FULL_COMBO = 2
+    GAME_PLAY_FLAG_PERFECT_COMBO = 3
+
+    # Chart type, as reported by profile save from the game
+    GAME_CHART_TYPE_NORMAL = 0
+    GAME_CHART_TYPE_HYPER = 1
+    GAME_CHART_TYPE_5_BUTTON = 2
+    GAME_CHART_TYPE_EX = 3
+    GAME_CHART_TYPE_BATTLE_NORMAL = 4
+    GAME_CHART_TYPE_BATTLE_HYPER = 5
+    GAME_CHART_TYPE_ENJOY_5_BUTTON = 6
+    GAME_CHART_TYPE_ENJOY_9_BUTTON = 7
+
+    # Extra chart types supported by Pop'n 19
+    CHART_TYPE_OLD_NORMAL = 4
+    CHART_TYPE_OLD_HYPER = 5
+    CHART_TYPE_OLD_EX = 6
+    CHART_TYPE_ENJOY_5_BUTTON = 7
+    CHART_TYPE_ENJOY_9_BUTTON = 8
+    CHART_TYPE_5_BUTTON = 9
+
+    # Chart type, as packed into a hiscore binary
+    GAME_CHART_TYPE_5_BUTTON_POSITION = 0
+    GAME_CHART_TYPE_NORMAL_POSITION = 1
+    GAME_CHART_TYPE_HYPER_POSITION = 2
+    GAME_CHART_TYPE_EX_POSITION = 3
+    GAME_CHART_TYPE_CHO_NORMAL_POSITION = 4
+    GAME_CHART_TYPE_CHO_HYPER_POSITION = 5
+    GAME_CHART_TYPE_CHO_EX_POSITION = 6
+
+    # Highest song ID we can represent
+    GAME_MAX_MUSIC_ID = 1045
+
+    def previous_version(self) -> Optional[PopnMusicBase]:
+        return PopnMusicSengokuRetsuden(self.data, self.config, self.model)
+
+    def __format_flags_for_score(self, score: Score) -> int:
+        # Format song flags (cleared/not, combo flags)
+        playedflag = {
+            self.CHART_TYPE_5_BUTTON: 0x2000,
+            self.CHART_TYPE_OLD_NORMAL: 0x0800,
+            self.CHART_TYPE_OLD_HYPER: 0x1000,
+            self.CHART_TYPE_OLD_EX: 0x4000,
+            self.CHART_TYPE_NORMAL: 0x0800,
+            self.CHART_TYPE_HYPER: 0x1000,
+            self.CHART_TYPE_EX: 0x4000,
+            # We don't have a played flag for these, only cleared/no play
+            self.CHART_TYPE_ENJOY_5_BUTTON: 0,
+            self.CHART_TYPE_ENJOY_9_BUTTON: 0,
+        }[score.chart]
+        # Shift value for cleared/failed/combo indicators
+        shift = {
+            self.CHART_TYPE_5_BUTTON: 4,
+            self.CHART_TYPE_OLD_NORMAL: 0,
+            self.CHART_TYPE_OLD_HYPER: 2,
+            self.CHART_TYPE_OLD_EX: 6,
+            self.CHART_TYPE_NORMAL: 0,
+            self.CHART_TYPE_HYPER: 2,
+            self.CHART_TYPE_EX: 6,
+            self.CHART_TYPE_ENJOY_5_BUTTON: 9,
+            self.CHART_TYPE_ENJOY_9_BUTTON: 8,
+        }[score.chart]
+        flags = {
+            self.PLAY_MEDAL_CIRCLE_FAILED: self.GAME_PLAY_FLAG_FAILED,
+            self.PLAY_MEDAL_DIAMOND_FAILED: self.GAME_PLAY_FLAG_FAILED,
+            self.PLAY_MEDAL_STAR_FAILED: self.GAME_PLAY_FLAG_FAILED,
+            self.PLAY_MEDAL_EASY_CLEAR: self.GAME_PLAY_FLAG_CLEARED,
+            self.PLAY_MEDAL_CIRCLE_CLEARED: self.GAME_PLAY_FLAG_CLEARED,
+            self.PLAY_MEDAL_DIAMOND_CLEARED: self.GAME_PLAY_FLAG_CLEARED,
+            self.PLAY_MEDAL_STAR_CLEARED: self.GAME_PLAY_FLAG_CLEARED,
+            self.PLAY_MEDAL_CIRCLE_FULL_COMBO: self.GAME_PLAY_FLAG_FULL_COMBO,
+            self.PLAY_MEDAL_DIAMOND_FULL_COMBO: self.GAME_PLAY_FLAG_FULL_COMBO,
+            self.PLAY_MEDAL_STAR_FULL_COMBO: self.GAME_PLAY_FLAG_FULL_COMBO,
+            self.PLAY_MEDAL_PERFECT: self.GAME_PLAY_FLAG_PERFECT_COMBO,
+        }[score.data.get_int('medal')]
+        return (flags << shift) | playedflag
+
+    def format_profile(self, userid: UserID, profile: ValidatedDict) -> Node:
+        root = Node.void('playerdata')
+
+        # Format profile
+        binary_profile = [0] * 2200
+
+        # Copy name
+        name_binary = profile.get_str('name', 'なし').encode('shift-jis')[0:12]
+        name_pos = 0
+        for byte in name_binary:
+            binary_profile[name_pos] = byte
+            name_pos = name_pos + 1
+
+        # Copy game mode
+        binary_profile[13] = {
+            0: 0,
+            1: 0,
+            2: 1,
+            3: 1,
+            4: 4,
+            5: 2,
+        }[profile.get_int('play_mode')]
+
+        # Copy miscelaneous values
+        binary_profile[16] = profile.get_int('last_play_flag') & 0xFF
+        binary_profile[44] = profile.get_int('option') & 0xFF
+        binary_profile[45] = (profile.get_int('option') >> 8) & 0xFF
+        binary_profile[46] = (profile.get_int('option') >> 16) & 0xFF
+        binary_profile[47] = (profile.get_int('option') >> 24) & 0xFF
+        binary_profile[60] = profile.get_int('chara') & 0xFF
+        binary_profile[61] = (profile.get_int('chara') >> 8) & 0xFF
+        binary_profile[62] = profile.get_int('music') & 0xFF
+        binary_profile[63] = (profile.get_int('music') >> 8) & 0xFF
+        binary_profile[64] = profile.get_int('sheet') & 0xFF
+        binary_profile[65] = profile.get_int('category') & 0xFF
+        binary_profile[67] = profile.get_int('medal_and_friend') & 0xFF
+
+        # Format Scores
+        hiscore_array = [0] * int((((self.GAME_MAX_MUSIC_ID * 7) * 17) + 7) / 8)
+        scores = self.data.remote.music.get_scores(self.game, self.version, userid)
+        for score in scores:
+            if score.id > self.GAME_MAX_MUSIC_ID:
+                continue
+
+            # Skip any scores for chart types we don't support
+            if score.chart in [
+                self.CHART_TYPE_EASY,
+            ]:
+                continue
+
+            flags = self.__format_flags_for_score(score)
+
+            flags_index = score.id * 2
+            binary_profile[108 + flags_index] = binary_profile[108 + flags_index] | (flags & 0xFF)
+            binary_profile[109 + flags_index] = binary_profile[109 + flags_index] | ((flags >> 8) & 0xFF)
+
+            if score.chart in [
+                self.CHART_TYPE_ENJOY_5_BUTTON,
+                self.CHART_TYPE_ENJOY_9_BUTTON,
+            ]:
+                # We don't return enjoy scores, just the flags that we played them
+                continue
+
+            # Format actual score, according to DB chart position
+            points = score.points
+
+            hiscore_index = (score.id * 7) + {
+                self.CHART_TYPE_5_BUTTON: self.GAME_CHART_TYPE_5_BUTTON_POSITION,
+                self.CHART_TYPE_OLD_NORMAL: self.GAME_CHART_TYPE_NORMAL_POSITION,
+                self.CHART_TYPE_OLD_HYPER: self.GAME_CHART_TYPE_HYPER_POSITION,
+                self.CHART_TYPE_OLD_EX: self.GAME_CHART_TYPE_EX_POSITION,
+                self.CHART_TYPE_NORMAL: self.GAME_CHART_TYPE_CHO_NORMAL_POSITION,
+                self.CHART_TYPE_HYPER: self.GAME_CHART_TYPE_CHO_HYPER_POSITION,
+                self.CHART_TYPE_EX: self.GAME_CHART_TYPE_CHO_EX_POSITION,
+            }[score.chart]
+            hiscore_byte_pos = int((hiscore_index * 17) / 8)
+            hiscore_bit_pos = int((hiscore_index * 17) % 8)
+            hiscore_value = points << hiscore_bit_pos
+            hiscore_array[hiscore_byte_pos] = hiscore_array[hiscore_byte_pos] | (hiscore_value & 0xFF)
+            hiscore_array[hiscore_byte_pos + 1] = hiscore_array[hiscore_byte_pos + 1] | ((hiscore_value >> 8) & 0xFF)
+            hiscore_array[hiscore_byte_pos + 2] = hiscore_array[hiscore_byte_pos + 2] | ((hiscore_value >> 16) & 0xFF)
+
+        # Format most played
+        most_played = [x[0] for x in self.data.local.music.get_most_played(self.game, self.version, userid, 20)]
+        while len(most_played) < 20:
+            most_played.append(-1)
+        profile_pos = 68
+        for musicid in most_played:
+            binary_profile[profile_pos] = musicid & 0xFF
+            binary_profile[profile_pos + 1] = (musicid >> 8) & 0xFF
+            profile_pos = profile_pos + 2
+
+        # Construct final profile
+        root.add_child(Node.binary('b', bytes(binary_profile)))
+        root.add_child(Node.binary('hiscore', bytes(hiscore_array)))
+        root.add_child(Node.binary('town', b''))
+
+        return root
+
+    def format_conversion(self, userid: UserID, profile: ValidatedDict) -> Node:
+        root = Node.void('playerdata')
+
+        root.add_child(Node.string('name', profile.get_str('name', 'なし')))
+        root.add_child(Node.s16('chara', profile.get_int('chara', -1)))
+        root.add_child(Node.s32('option', profile.get_int('option', 0)))
+        root.add_child(Node.u8('version', 0))
+        root.add_child(Node.u8('kind', 0))
+        root.add_child(Node.u8('season', 0))
+
+        medals = [0] * (self.GAME_MAX_MUSIC_ID)
+        scores = self.data.remote.music.get_scores(self.game, self.version, userid)
+        for score in scores:
+            if score.id > self.GAME_MAX_MUSIC_ID:
+                continue
+
+            # Skip any scores for chart types we don't support
+            if score.chart in [
+                self.CHART_TYPE_EASY,
+            ]:
+                continue
+
+            flags = self.__format_flags_for_score(score)
+            medals[score.id] = medals[score.id] | flags
+        root.add_child(Node.u16_array('clear_medal', medals))
+
+        return root
+
+    def unformat_profile(self, userid: UserID, request: Node, oldprofile: ValidatedDict) -> ValidatedDict:
+        newprofile = copy.deepcopy(oldprofile)
+
+        # Extract the playmode, important for scores later
+        playmode = int(request.attribute('play_mode'))
+        newprofile.replace_int('play_mode', playmode)
+
+        # Extract profile options
+        newprofile.replace_int('chara', int(request.attribute('chara_num')))
+        if 'option' in request.attributes:
+            newprofile.replace_int('option', int(request.attribute('option')))
+        if 'last_play_flag' in request.attributes:
+            newprofile.replace_int('last_play_flag', int(request.attribute('last_play_flag')))
+        if 'medal_and_friend' in request.attributes:
+            newprofile.replace_int('medal_and_friend', int(request.attribute('medal_and_friend')))
+        if 'music_num' in request.attributes:
+            newprofile.replace_int('music', int(request.attribute('music_num')))
+        if 'sheet_num' in request.attributes:
+            newprofile.replace_int('sheet', int(request.attribute('sheet_num')))
+        if 'category_num' in request.attributes:
+            newprofile.replace_int('category', int(request.attribute('category_num')))
+
+        # Keep track of play statistics
+        self.update_play_statistics(userid)
+
+        # Extract scores
+        for node in request.children:
+            if node.name == 'music':
+                songid = int(node.attribute('music_num'))
+                chart = int(node.attribute('sheet_num'))
+                points = int(node.attribute('score'))
+                data = int(node.attribute('data'))
+
+                # We never save battle scores
+                if chart in [
+                    self.GAME_CHART_TYPE_BATTLE_NORMAL,
+                    self.GAME_CHART_TYPE_BATTLE_HYPER,
+                ]:
+                    continue
+
+                # Arrange order to be compatible with future mixes
+                if playmode == self.GAME_PLAY_MODE_CHO_CHALLENGE:
+                    if chart in [
+                        self.GAME_CHART_TYPE_5_BUTTON,
+                        self.GAME_CHART_TYPE_ENJOY_5_BUTTON,
+                        self.GAME_CHART_TYPE_ENJOY_9_BUTTON,
+                    ]:
+                        # We don't save 5 button for cho scores, or enjoy modes
+                        continue
+                    chart = {
+                        self.GAME_CHART_TYPE_NORMAL: self.CHART_TYPE_NORMAL,
+                        self.GAME_CHART_TYPE_HYPER: self.CHART_TYPE_HYPER,
+                        self.GAME_CHART_TYPE_EX: self.CHART_TYPE_EX,
+                    }[chart]
+                else:
+                    chart = {
+                        self.GAME_CHART_TYPE_NORMAL: self.CHART_TYPE_OLD_NORMAL,
+                        self.GAME_CHART_TYPE_HYPER: self.CHART_TYPE_OLD_HYPER,
+                        self.GAME_CHART_TYPE_5_BUTTON: self.CHART_TYPE_5_BUTTON,
+                        self.GAME_CHART_TYPE_EX: self.CHART_TYPE_OLD_EX,
+                        self.GAME_CHART_TYPE_ENJOY_5_BUTTON: self.CHART_TYPE_ENJOY_5_BUTTON,
+                        self.GAME_CHART_TYPE_ENJOY_9_BUTTON: self.CHART_TYPE_ENJOY_9_BUTTON,
+                    }[chart]
+
+                # Extract play flags
+                shift = {
+                    self.CHART_TYPE_5_BUTTON: 4,
+                    self.CHART_TYPE_OLD_NORMAL: 0,
+                    self.CHART_TYPE_OLD_HYPER: 2,
+                    self.CHART_TYPE_OLD_EX: 6,
+                    self.CHART_TYPE_NORMAL: 0,
+                    self.CHART_TYPE_HYPER: 2,
+                    self.CHART_TYPE_EX: 6,
+                    self.CHART_TYPE_ENJOY_5_BUTTON: 9,
+                    self.CHART_TYPE_ENJOY_9_BUTTON: 8,
+                }[chart]
+
+                if chart in [
+                    self.CHART_TYPE_ENJOY_5_BUTTON,
+                    self.CHART_TYPE_ENJOY_9_BUTTON,
+                ]:
+                    # We only store cleared or not played for enjoy mode
+                    mask = 0x1
+                else:
+                    # We store all data for regular charts
+                    mask = 0x3
+
+                # Grab flags, map to medals in DB. Choose lowest one for each so
+                # a newer pop'n can still improve scores and medals.
+                flags = (data >> shift) & mask
+                medal = {
+                    self.GAME_PLAY_FLAG_FAILED: self.PLAY_MEDAL_CIRCLE_FAILED,
+                    self.GAME_PLAY_FLAG_CLEARED: self.PLAY_MEDAL_CIRCLE_CLEARED,
+                    self.GAME_PLAY_FLAG_FULL_COMBO: self.PLAY_MEDAL_CIRCLE_FULL_COMBO,
+                    self.GAME_PLAY_FLAG_PERFECT_COMBO: self.PLAY_MEDAL_PERFECT,
+                }[flags]
+                self.update_score(userid, songid, chart, points, medal)
+
+        return newprofile
+
+    def handle_game_request(self, request: Node) -> Optional[Node]:
+        method = request.attribute('method')
+
+        if method == 'get':
+            # TODO: Hook these up to config so we can change this
+            root = Node.void('game')
+            root.set_attribute('game_phase', '2')
+            root.set_attribute('psp_phase', '2')
+            return root
+
+        if method == 'active':
+            # Update the name of this cab for admin purposes
+            self.update_machine_name(request.attribute('shop_name'))
+            return Node.void('game')
+
+        if method == 'taxphase':
+            return Node.void('game')
+
+        # Invalid method
+        return None
+
+    def handle_playerdata_request(self, request: Node) -> Optional[Node]:
+        method = request.attribute('method')
+
+        if method == 'expire':
+            return Node.void('playerdata')
+
+        elif method == 'logout':
+            return Node.void('playerdata')
+
+        elif method == 'get':
+            modelstring = request.attribute('model')
+            refid = request.attribute('ref_id')
+            root = self.get_profile_by_refid(
+                refid,
+                self.NEW_PROFILE_ONLY if modelstring is None else self.OLD_PROFILE_ONLY,
+            )
+            if root is None:
+                root = Node.void('playerdata')
+                root.set_attribute('status', str(Status.NO_PROFILE))
+            return root
+
+        elif method == 'new':
+            refid = request.attribute('ref_id')
+            name = request.attribute('name')
+            root = self.new_profile_by_refid(refid, name)
+            if root is None:
+                root = Node.void('playerdata')
+                root.set_attribute('status', str(Status.NO_PROFILE))
+            return root
+
+        elif method == 'set':
+            refid = request.attribute('ref_id')
+
+            root = Node.void('playerdata')
+            if refid is None:
+                return root
+
+            userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+            if userid is None:
+                return root
+
+            oldprofile = self.get_profile(userid) or ValidatedDict()
+            newprofile = self.unformat_profile(userid, request, oldprofile)
+
+            if newprofile is not None:
+                self.put_profile(userid, newprofile)
+
+            return root
+
+        # Invalid method
+        return None

bemani/backend/reflec/__init__.py

@@ -0,0 +1,2 @@
+from bemani.backend.reflec.factory import ReflecBeatFactory
+from bemani.backend.reflec.base import ReflecBeatBase

bemani/backend/reflec/base.py

@@ -0,0 +1,276 @@
+# vim: set fileencoding=utf-8
+from typing import Dict, List, Optional
+
+from bemani.backend.base import Base
+from bemani.backend.core import CoreHandler, CardManagerHandler, PASELIHandler
+from bemani.common import ValidatedDict, GameConstants, DBConstants, Time
+from bemani.data import Machine, ScoreSaveException, UserID
+from bemani.protocol import Node
+
+
+class ReflecBeatBase(CoreHandler, CardManagerHandler, PASELIHandler, Base):
+    """
+    Base game class for all Reflec Beat version that we support.
+    """
+
+    game = GameConstants.REFLEC_BEAT
+
+    # Chart types, as stored in the DB
+    CHART_TYPE_BASIC = 0
+    CHART_TYPE_MEDIUM = 1
+    CHART_TYPE_HARD = 2
+    CHART_TYPE_SPECIAL = 3
+
+    # Clear types, as saved/loaded from the DB
+    CLEAR_TYPE_NO_PLAY = DBConstants.REFLEC_BEAT_CLEAR_TYPE_NO_PLAY
+    CLEAR_TYPE_FAILED = DBConstants.REFLEC_BEAT_CLEAR_TYPE_FAILED
+    CLEAR_TYPE_CLEARED = DBConstants.REFLEC_BEAT_CLEAR_TYPE_CLEARED
+    CLEAR_TYPE_HARD_CLEARED = DBConstants.REFLEC_BEAT_CLEAR_TYPE_HARD_CLEARED
+    CLEAR_TYPE_S_HARD_CLEARED = DBConstants.REFLEC_BEAT_CLEAR_TYPE_S_HARD_CLEARED
+
+    # Combo types, as saved/loaded from the DB
+    COMBO_TYPE_NONE = DBConstants.REFLEC_BEAT_COMBO_TYPE_NONE
+    COMBO_TYPE_ALMOST_COMBO = DBConstants.REFLEC_BEAT_COMBO_TYPE_ALMOST_COMBO
+    COMBO_TYPE_FULL_COMBO = DBConstants.REFLEC_BEAT_COMBO_TYPE_FULL_COMBO
+    COMBO_TYPE_FULL_COMBO_ALL_JUST = DBConstants.REFLEC_BEAT_COMBO_TYPE_FULL_COMBO_ALL_JUST
+
+    def previous_version(self) -> Optional['ReflecBeatBase']:
+        """
+        Returns the previous version of the game, based on this game. Should
+        be overridden.
+        """
+        return None
+
+    def extra_services(self) -> List[str]:
+        """
+        Return the local2 and lobby2 service so that matching will work on newer
+        Reflec Beat games.
+        """
+        return [
+            'local2',
+            'lobby2',
+        ]
+
+    def format_profile(self, userid: UserID, profile: ValidatedDict) -> Node:
+        """
+        Base handler for a profile. Given a userid and a profile dictionary,
+        return a Node representing a profile. Should be overridden.
+        """
+        return Node.void('pc')
+
+    def unformat_profile(self, userid: UserID, request: Node, oldprofile: ValidatedDict) -> ValidatedDict:
+        """
+        Base handler for profile parsing. Given a request and an old profile,
+        return a new profile that's been updated with the contents of the request.
+        Should be overridden.
+        """
+        return oldprofile
+
+    def get_profile_by_refid(self, refid: Optional[str]) -> Optional[Node]:
+        """
+        Given a RefID, return a formatted profile node. Basically every game
+        needs a profile lookup, even if it handles where that happens in
+        a different request. This is provided for code deduplication.
+        """
+        if refid is None:
+            return None
+
+        userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        if userid is None:
+            # User doesn't exist but should at this point
+            return None
+
+        # Trying to import from current version
+        profile = self.get_profile(userid)
+        if profile is None:
+            return None
+        return self.format_profile(userid, profile)
+
+    def put_profile_by_refid(self, refid: Optional[str], request: Node) -> Optional[ValidatedDict]:
+        """
+        Given a RefID and a request node, unformat the profile and save it.
+        """
+        userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        if userid is None:
+            return None
+
+        oldprofile = self.get_profile(userid)
+        if oldprofile is None:
+            # Create one so we can get refid/extid
+            self.put_profile(userid, ValidatedDict())
+            oldprofile = self.get_profile(userid)
+        newprofile = self.unformat_profile(userid, request, oldprofile)
+        if newprofile is not None:
+            self.put_profile(userid, newprofile)
+            return newprofile
+        else:
+            return oldprofile
+
+    def get_machine_by_id(self, shop_id: int) -> Optional[Machine]:
+        pcbid = self.data.local.machine.from_machine_id(shop_id)
+        if pcbid is not None:
+            return self.data.local.machine.get_machine(pcbid)
+        else:
+            return None
+
+    def update_score(
+        self,
+        userid: UserID,
+        songid: int,
+        chart: int,
+        points: int,
+        achievement_rate: int,
+        clear_type: int,
+        combo_type: int,
+        miss_count: int,
+        combo: Optional[int]=None,
+        stats: Optional[Dict[str, int]]=None,
+        param: Optional[int]=None,
+        kflag: Optional[int]=None,
+    ) -> None:
+        """
+        Given various pieces of a score, update the user's high score and score
+        history in a controlled manner, so all games in Reflec series can expect
+        the same attributes in a score. Note that the clear_types passed here are
+        expected to be converted from game identifier to our internal identifier,
+        so that any game in the series may convert them back.
+        """
+        # Range check clear type
+        if clear_type not in [
+            self.CLEAR_TYPE_NO_PLAY,
+            self.CLEAR_TYPE_FAILED,
+            self.CLEAR_TYPE_CLEARED,
+            self.CLEAR_TYPE_HARD_CLEARED,
+            self.CLEAR_TYPE_S_HARD_CLEARED,
+        ]:
+            raise Exception("Invalid clear_type value {}".format(clear_type))
+
+        # Range check combo type
+        if combo_type not in [
+            self.COMBO_TYPE_NONE,
+            self.COMBO_TYPE_ALMOST_COMBO,
+            self.COMBO_TYPE_FULL_COMBO,
+            self.COMBO_TYPE_FULL_COMBO_ALL_JUST,
+        ]:
+            raise Exception("Invalid combo_type value {}".format(combo_type))
+
+        oldscore = self.data.local.music.get_score(
+            self.game,
+            self.version,
+            userid,
+            songid,
+            chart,
+        )
+
+        # Score history is verbatum, instead of highest score
+        now = Time.now()
+        history = ValidatedDict({})
+        oldpoints = points
+
+        if oldscore is None:
+            # If it is a new score, create a new dictionary to add to
+            scoredata = ValidatedDict({})
+            highscore = True
+        else:
+            # Set the score to any new record achieved
+            highscore = points >= oldscore.points
+            points = max(points, oldscore.points)
+            scoredata = oldscore.data
+
+        # Update the last played time
+        scoredata.replace_int('last_played_time', now)
+
+        # Replace clear type with highest value and timestamps
+        if clear_type >= scoredata.get_int('clear_type'):
+            scoredata.replace_int('clear_type', max(scoredata.get_int('clear_type'), clear_type))
+            scoredata.replace_int('best_clear_type_time', now)
+        history.replace_int('clear_type', clear_type)
+
+        # Replace combo type with highest value and timestamps
+        if combo_type >= scoredata.get_int('combo_type'):
+            scoredata.replace_int('combo_type', max(scoredata.get_int('combo_type'), combo_type))
+            scoredata.replace_int('best_clear_type_time', now)
+        history.replace_int('combo_type', combo_type)
+
+        # Update the combo for this song
+        if combo is not None:
+            scoredata.replace_int('combo', max(scoredata.get_int('combo'), combo))
+            history.replace_int('combo', combo)
+
+        # Update the param for this song
+        if param is not None:
+            scoredata.replace_int('param', max(scoredata.get_int('param'), param))
+            history.replace_int('param', param)
+
+        # Update the kflag for this song
+        if kflag is not None:
+            scoredata.replace_int('kflag', max(scoredata.get_int('kflag'), kflag))
+            history.replace_int('kflag', kflag)
+
+        # Update win/lost/draw stats for this song
+        if stats is not None:
+            scoredata.replace_dict('stats', stats)
+            history.replace_dict('stats', stats)
+
+        # Update the achievement rate with timestamps
+        if achievement_rate >= scoredata.get_int('achievement_rate'):
+            scoredata.replace_int('achievement_rate', max(scoredata.get_int('achievement_rate'), achievement_rate))
+            scoredata.replace_int('best_achievement_rate_time', now)
+        history.replace_int('achievement_rate', achievement_rate)
+
+        # Update the miss count with timestamps, either if it was lowered, or if the old value was blank.
+        # If the new value is -1 (we didn't get a miss count this time), never update the old value.
+        if miss_count >= 0:
+            if miss_count <= scoredata.get_int('miss_count', 999999) or scoredata.get_int('miss_count') == -1:
+                scoredata.replace_int('miss_count', min(scoredata.get_int('miss_count', 999999), miss_count))
+                scoredata.replace_int('best_miss_count_time', now)
+        history.replace_int('miss_count', miss_count)
+
+        # Look up where this score was earned
+        lid = self.get_machine_id()
+
+        # Reflec Beat happens to send all songs that were played by a player
+        # at the end of the round. It sends timestamps for the songs, but as of
+        # Colette they were identical for each song in the round. So, if a user
+        # plays the same song/chart# more than once in a round, we will end up
+        # failing to store the attempt since we don't allow two of the same
+        # attempt at the same time for the same user and song/chart. So, bump
+        # the timestamp by one second and retry well past the maximum number of
+        # songs.
+        for bump in range(10):
+            timestamp = now + bump
+
+            # Write the new score back
+            self.data.local.music.put_score(
+                self.game,
+                self.version,
+                userid,
+                songid,
+                chart,
+                lid,
+                points,
+                scoredata,
+                highscore,
+                timestamp=timestamp,
+            )
+
+            try:
+                # Save the history of this score too
+                self.data.local.music.put_attempt(
+                    self.game,
+                    self.version,
+                    userid,
+                    songid,
+                    chart,
+                    lid,
+                    oldpoints,
+                    history,
+                    highscore,
+                    timestamp=timestamp,
+                )
+            except ScoreSaveException:
+                # Try again one second in the future
+                continue
+
+            # We saved successfully
+            break

bemani/backend/reflec/factory.py

@@ -0,0 +1,81 @@
+from typing import Any, Dict, List, Optional, Type
+
+from bemani.backend.base import Base, Factory
+from bemani.backend.reflec.reflecbeat import ReflecBeat
+from bemani.backend.reflec.limelight import ReflecBeatLimelight
+from bemani.backend.reflec.colette import ReflecBeatColette
+from bemani.backend.reflec.groovin import ReflecBeatGroovin
+from bemani.backend.reflec.volzza import ReflecBeatVolzza
+from bemani.backend.reflec.volzza2 import ReflecBeatVolzza2
+from bemani.common import Model, VersionConstants
+from bemani.data import Data
+
+
+class ReflecBeatFactory(Factory):
+
+    MANAGED_CLASSES: List[Type[Base]] = [
+        ReflecBeat,
+        ReflecBeatLimelight,
+        ReflecBeatColette,
+        ReflecBeatGroovin,
+        ReflecBeatVolzza,
+        ReflecBeatVolzza2,
+    ]
+
+    @classmethod
+    def register_all(cls) -> None:
+        for game in ['KBR', 'LBR', 'MBR']:
+            Base.register(game, ReflecBeatFactory)
+
+    @classmethod
+    def create(cls, data: Data, config: Dict[str, Any], model: Model, parentmodel: Optional[Model]=None) -> Optional[Base]:
+
+        def version_from_date(date: int) -> Optional[int]:
+            if date < 2014060400:
+                return VersionConstants.REFLEC_BEAT_COLETTE
+            if date >= 2014060400 and date < 2015102800:
+                return VersionConstants.REFLEC_BEAT_GROOVIN
+            if date >= 2015102800 and date < 2016032400:
+                return VersionConstants.REFLEC_BEAT_VOLZZA
+            if date >= 2016032400 and date < 2016120100:
+                return VersionConstants.REFLEC_BEAT_VOLZZA_2
+            if date >= 2016120100:
+                return VersionConstants.REFLEC_BEAT_REFLESIA
+            return None
+
+        if model.game == 'KBR':
+            return ReflecBeat(data, config, model)
+        if model.game == 'LBR':
+            return ReflecBeatLimelight(data, config, model)
+        if model.game == 'MBR':
+            if model.version is None:
+                if parentmodel is None:
+                    return None
+
+                if parentmodel.game not in ['KBR', 'LBR', 'MBR']:
+                    return None
+                parentversion = version_from_date(parentmodel.version)
+                if parentversion == VersionConstants.REFLEC_BEAT_COLETTE:
+                    return ReflecBeatLimelight(data, config, model)
+                if parentversion == VersionConstants.REFLEC_BEAT_GROOVIN:
+                    return ReflecBeatColette(data, config, model)
+                if parentversion == VersionConstants.REFLEC_BEAT_VOLZZA:
+                    return ReflecBeatGroovin(data, config, model)
+                if parentversion == VersionConstants.REFLEC_BEAT_VOLZZA_2:
+                    return ReflecBeatVolzza(data, config, model)
+
+                # Unknown older version
+                return None
+
+            version = version_from_date(model.version)
+            if version == VersionConstants.REFLEC_BEAT_COLETTE:
+                return ReflecBeatColette(data, config, model)
+            if version == VersionConstants.REFLEC_BEAT_GROOVIN:
+                return ReflecBeatGroovin(data, config, model)
+            if version == VersionConstants.REFLEC_BEAT_VOLZZA:
+                return ReflecBeatVolzza(data, config, model)
+            if version == VersionConstants.REFLEC_BEAT_VOLZZA_2:
+                return ReflecBeatVolzza2(data, config, model)
+
+        # Unknown game version
+        return None

bemani/backend/reflec/limelight.py

@@ -0,0 +1,960 @@
+import copy
+from typing import Optional, Dict, Any, Tuple
+
+from bemani.backend.reflec.base import ReflecBeatBase
+from bemani.backend.reflec.reflecbeat import ReflecBeat
+
+from bemani.common import ValidatedDict, VersionConstants, ID, Time
+from bemani.data import UserID
+from bemani.protocol import Node
+
+
+class ReflecBeatLimelight(ReflecBeatBase):
+
+    name = "REFLEC BEAT limelight"
+    version = VersionConstants.REFLEC_BEAT_LIMELIGHT
+
+    # Clear types according to the game
+    GAME_CLEAR_TYPE_NO_PLAY = 0
+    GAME_CLEAR_TYPE_FAILED = 2
+    GAME_CLEAR_TYPE_CLEARED = 3
+    GAME_CLEAR_TYPE_FULL_COMBO = 4
+
+    def previous_version(self) -> Optional[ReflecBeatBase]:
+        return ReflecBeat(self.data, self.config, self.model)
+
+    @classmethod
+    def get_settings(cls) -> Dict[str, Any]:
+        """
+        Return all of our front-end modifiably settings.
+        """
+        return {
+            'bools': [
+                {
+                    'name': 'Force Song Unlock',
+                    'tip': 'Force unlock all songs.',
+                    'category': 'game_config',
+                    'setting': 'force_unlock_songs',
+                },
+            ],
+            'ints': [],
+        }
+
+    def __db_to_game_clear_type(self, db_clear_type: int, db_combo_type: int) -> int:
+        if db_clear_type == self.CLEAR_TYPE_NO_PLAY:
+            return self.GAME_CLEAR_TYPE_NO_PLAY
+        if db_clear_type == self.CLEAR_TYPE_FAILED:
+            return self.GAME_CLEAR_TYPE_FAILED
+        if db_clear_type in [
+            self.CLEAR_TYPE_CLEARED,
+            self.CLEAR_TYPE_HARD_CLEARED,
+            self.CLEAR_TYPE_S_HARD_CLEARED,
+        ]:
+            if db_combo_type in [
+                self.COMBO_TYPE_NONE,
+                self.COMBO_TYPE_ALMOST_COMBO,
+            ]:
+                return self.GAME_CLEAR_TYPE_CLEARED
+            if db_combo_type in [
+                self.COMBO_TYPE_FULL_COMBO,
+                self.COMBO_TYPE_FULL_COMBO_ALL_JUST,
+            ]:
+                return self.GAME_CLEAR_TYPE_FULL_COMBO
+
+            raise Exception('Invalid db_combo_type {}'.format(db_combo_type))
+        raise Exception('Invalid db_clear_type {}'.format(db_clear_type))
+
+    def __game_to_db_clear_type(self, game_clear_type: int) -> Tuple[int, int]:
+        if game_clear_type == self.GAME_CLEAR_TYPE_NO_PLAY:
+            return (self.CLEAR_TYPE_NO_PLAY, self.COMBO_TYPE_NONE)
+        if game_clear_type == self.GAME_CLEAR_TYPE_FAILED:
+            return (self.CLEAR_TYPE_FAILED, self.COMBO_TYPE_NONE)
+        if game_clear_type == self.GAME_CLEAR_TYPE_CLEARED:
+            return (self.CLEAR_TYPE_CLEARED, self.COMBO_TYPE_NONE)
+        if game_clear_type == self.GAME_CLEAR_TYPE_FULL_COMBO:
+            return (self.CLEAR_TYPE_CLEARED, self.COMBO_TYPE_FULL_COMBO)
+
+        raise Exception('Invalid game_clear_type {}'.format(game_clear_type))
+
+    def handle_log_exception_request(self, request: Node) -> Node:
+        return Node.void('log')
+
+    def handle_log_pcb_status_request(self, request: Node) -> Node:
+        return Node.void('log')
+
+    def handle_log_opsetting_request(self, request: Node) -> Node:
+        return Node.void('log')
+
+    def handle_log_play_request(self, request: Node) -> Node:
+        return Node.void('log')
+
+    def handle_pcbinfo_get_request(self, request: Node) -> Node:
+        shop_id = ID.parse_machine_id(request.child_value('lid'))
+        machine = self.get_machine_by_id(shop_id)
+        if machine is not None:
+            machine_name = machine.name
+            close = machine.data.get_bool('close')
+            hour = machine.data.get_int('hour')
+            minute = machine.data.get_int('minute')
+            pref = machine.data.get_int('pref', 51)
+        else:
+            machine_name = ''
+            close = False
+            hour = 0
+            minute = 0
+            pref = 51
+
+        root = Node.void('pcbinfo')
+        info = Node.void('info')
+        root.add_child(info)
+
+        info.add_child(Node.string('name', machine_name))
+        info.add_child(Node.s16('pref', pref))
+        info.add_child(Node.bool('close', close))
+        info.add_child(Node.u8('hour', hour))
+        info.add_child(Node.u8('min', minute))
+
+        return root
+
+    def handle_pcbinfo_set_request(self, request: Node) -> Node:
+        self.update_machine_name(request.child_value('info/name'))
+        self.update_machine_data({
+            'close': request.child_value('info/close'),
+            'hour': request.child_value('info/hour'),
+            'minute': request.child_value('info/min'),
+            'pref': request.child_value('info/pref'),
+        })
+        return Node.void('pcbinfo')
+
+    def __add_event_info(self, request: Node) -> None:
+        events: Dict[int, int] = {}
+
+        for (eventid, phase) in events.items():
+            data = Node.void('data')
+            request.add_child(data)
+            data.add_child(Node.s32('type', -1))
+            data.add_child(Node.s32('value', -1))
+
+    def handle_sysinfo_get_request(self, request: Node) -> Node:
+        root = Node.void('sysinfo')
+        trd = Node.void('trd')
+        root.add_child(trd)
+
+        # Add event info
+        self.__add_event_info(trd)
+
+        return root
+
+    def handle_ranking_read_request(self, request: Node) -> Node:
+        root = Node.void('ranking')
+
+        licenses = Node.void('lic_10')
+        root.add_child(licenses)
+        originals = Node.void('org_10')
+        root.add_child(originals)
+
+        licenses.add_child(Node.time('time', Time.now()))
+        originals.add_child(Node.time('time', Time.now()))
+
+        hitchart = self.data.local.music.get_hit_chart(self.game, self.version, 10)
+        rank = 1
+        for (mid, plays) in hitchart:
+            record = Node.void('record')
+            originals.add_child(record)
+            record.add_child(Node.s16('id', mid))
+            record.add_child(Node.s16('rank', rank))
+            rank = rank + 1
+
+        return root
+
+    def handle_event_r_get_all_request(self, request: Node) -> Node:
+        limit = request.child_value('limit')
+
+        comments = [
+            achievement for achievement in
+            self.data.local.user.get_all_time_based_achievements(self.game, self.version)
+            if achievement[1].type == 'puzzle_comment'
+        ]
+        comments.sort(key=lambda x: x[1].timestamp, reverse=True)
+        statuses = self.data.local.lobby.get_all_play_session_infos(self.game, self.version)
+        statuses.sort(key=lambda x: x[1]['time'], reverse=True)
+
+        # Cap all comment blocks to the limit
+        if limit >= 0:
+            comments = comments[:limit]
+            statuses = statuses[:limit]
+
+        # Mapping of profiles to userIDs
+        uid_mapping = {
+            uid: prof for (uid, prof) in self.get_any_profiles(
+                [c[0] for c in comments] +
+                [s[0] for s in statuses]
+            )
+        }
+
+        # Mapping of location ID to machine name
+        lid_mapping: Dict[int, str] = {}
+
+        root = Node.void('event_r')
+        root.add_child(Node.s32('time', Time.now()))
+        statusnode = Node.void('status')
+        root.add_child(statusnode)
+        commentnode = Node.void('comment')
+        root.add_child(commentnode)
+
+        for (uid, comment) in comments:
+            lid = ID.parse_machine_id(comment.data.get_str('lid'))
+
+            # Look up external data for the request
+            if lid not in lid_mapping:
+                machine = self.get_machine_by_id(lid)
+                if machine is not None:
+                    lid_mapping[lid] = machine.name
+                else:
+                    lid_mapping[lid] = ''
+
+            c = Node.void('c')
+            commentnode.add_child(c)
+            c.add_child(Node.s32('uid', uid_mapping[uid].get_int('extid')))
+            c.add_child(Node.string('p_name', uid_mapping[uid].get_str('name')))
+            c.add_child(Node.s32('exp', uid_mapping[uid].get_int('exp')))
+            c.add_child(Node.s32('customize', comment.data.get_int('customize')))
+            c.add_child(Node.s32('tid', comment.data.get_int('teamid')))
+            c.add_child(Node.string('t_name', comment.data.get_str('teamname')))
+            c.add_child(Node.string('lid', comment.data.get_str('lid')))
+            c.add_child(Node.string('s_name', lid_mapping[lid]))
+            c.add_child(Node.s8('pref', comment.data.get_int('prefecture')))
+            c.add_child(Node.s32('time', comment.timestamp))
+            c.add_child(Node.string('comment', comment.data.get_str('comment')))
+            c.add_child(Node.bool('is_tweet', comment.data.get_bool('tweet')))
+
+        for (uid, status) in statuses:
+            lid = ID.parse_machine_id(status.get_str('lid'))
+
+            # Look up external data for the request
+            if lid not in lid_mapping:
+                machine = self.get_machine_by_id(lid)
+                if machine is not None:
+                    lid_mapping[lid] = machine.name
+                else:
+                    lid_mapping[lid] = ''
+
+            s = Node.void('s')
+            statusnode.add_child(s)
+            s.add_child(Node.s32('uid', uid_mapping[uid].get_int('extid')))
+            s.add_child(Node.string('p_name', uid_mapping[uid].get_str('name')))
+            s.add_child(Node.s32('exp', uid_mapping[uid].get_int('exp')))
+            s.add_child(Node.s32('customize', status.get_int('customize')))
+            s.add_child(Node.s32('tid', uid_mapping[uid].get_int('team_id', -1)))
+            s.add_child(Node.string('t_name', uid_mapping[uid].get_str('team_name', '')))
+            s.add_child(Node.string('lid', status.get_str('lid')))
+            s.add_child(Node.string('s_name', lid_mapping[lid]))
+            s.add_child(Node.s8('pref', status.get_int('prefecture')))
+            s.add_child(Node.s32('time', status.get_int('time')))
+            s.add_child(Node.s8('status', status.get_int('status')))
+            s.add_child(Node.s8('stage', status.get_int('stage')))
+            s.add_child(Node.s32('mid', status.get_int('mid')))
+            s.add_child(Node.s8('ng', status.get_int('ng')))
+
+        return root
+
+    def handle_event_w_add_comment_request(self, request: Node) -> Node:
+        extid = request.child_value('uid')
+        userid = self.data.remote.user.from_extid(self.game, self.version, extid)
+        if userid is None:
+            # Anonymous comment
+            userid = UserID(0)
+
+        customize = request.child_value('customize')
+        lid = request.child_value('lid')
+        teamid = request.child_value('tid')
+        teamname = request.child_value('t_name')
+        prefecture = request.child_value('pref')
+        comment = request.child_value('comment')
+        is_tweet = request.child_value('is_tweet')
+
+        # Link comment to user's profile
+        self.data.local.user.put_time_based_achievement(
+            self.game,
+            self.version,
+            userid,
+            0,  # We never have an ID for this, since comments are add-only
+            'puzzle_comment',
+            {
+                'customize': customize,
+                'lid': lid,
+                'teamid': teamid,
+                'teamname': teamname,
+                'prefecture': prefecture,
+                'comment': comment,
+                'tweet': is_tweet,
+            },
+        )
+
+        return Node.void('event_w')
+
+    def handle_event_w_update_status_request(self, request: Node) -> Node:
+        # Update user status so puzzle comments can show it
+        extid = request.child_value('uid')
+        userid = self.data.remote.user.from_extid(self.game, self.version, extid)
+        if userid is not None:
+            customize = request.child_value('customize')
+            status = request.child_value('status')
+            stage = request.child_value('stage')
+            mid = request.child_value('mid')
+            ng = request.child_value('ng')
+            lid = request.child_value('lid')
+            prefecture = request.child_value('pref')
+
+            self.data.local.lobby.put_play_session_info(
+                self.game,
+                self.version,
+                userid,
+                {
+                    'customize': customize,
+                    'status': status,
+                    'stage': stage,
+                    'mid': mid,
+                    'ng': ng,
+                    'lid': lid,
+                    'prefecture': prefecture,
+                },
+            )
+        return Node.void('event_w')
+
+    def handle_lobby_entry_request(self, request: Node) -> Node:
+        root = Node.void('lobby')
+        root.add_child(Node.s32('interval', 120))
+        root.add_child(Node.s32('interval_p', 120))
+
+        # Create a lobby entry for this user
+        extid = request.child_value('e/uid')
+        userid = self.data.remote.user.from_extid(self.game, self.version, extid)
+        if userid is not None:
+            profile = self.get_profile(userid)
+            self.data.local.lobby.put_lobby(
+                self.game,
+                self.version,
+                userid,
+                {
+                    'mid': request.child_value('e/mid'),
+                    'ng': request.child_value('e/ng'),
+                    'mopt': request.child_value('e/mopt'),
+                    'tid': request.child_value('e/tid'),
+                    'tn': request.child_value('e/tn'),
+                    'topt': request.child_value('e/topt'),
+                    'lid': request.child_value('e/lid'),
+                    'sn': request.child_value('e/sn'),
+                    'pref': request.child_value('e/pref'),
+                    'stg': request.child_value('e/stg'),
+                    'pside': request.child_value('e/pside'),
+                    'eatime': request.child_value('e/eatime'),
+                    'ga': request.child_value('e/ga'),
+                    'gp': request.child_value('e/gp'),
+                    'la': request.child_value('e/la'),
+                }
+            )
+            lobby = self.data.local.lobby.get_lobby(
+                self.game,
+                self.version,
+                userid,
+            )
+            root.add_child(Node.s32('eid', lobby.get_int('id')))
+            e = Node.void('e')
+            root.add_child(e)
+            e.add_child(Node.s32('eid', lobby.get_int('id')))
+            e.add_child(Node.u16('mid', lobby.get_int('mid')))
+            e.add_child(Node.u8('ng', lobby.get_int('ng')))
+            e.add_child(Node.s32('uid', profile.get_int('extid')))
+            e.add_child(Node.string('pn', profile.get_str('name')))
+            e.add_child(Node.s32('uattr', profile.get_int('uattr')))
+            e.add_child(Node.s32('mopt', lobby.get_int('mopt')))
+            e.add_child(Node.s16('mg', profile.get_int('mg')))
+            e.add_child(Node.s32('tid', lobby.get_int('tid')))
+            e.add_child(Node.string('tn', lobby.get_str('tn')))
+            e.add_child(Node.s32('topt', lobby.get_int('topt')))
+            e.add_child(Node.string('lid', lobby.get_str('lid')))
+            e.add_child(Node.string('sn', lobby.get_str('sn')))
+            e.add_child(Node.u8('pref', lobby.get_int('pref')))
+            e.add_child(Node.s8('stg', lobby.get_int('stg')))
+            e.add_child(Node.s8('pside', lobby.get_int('pside')))
+            e.add_child(Node.s16('eatime', lobby.get_int('eatime')))
+            e.add_child(Node.u8_array('ga', lobby.get_int_array('ga', 4)))
+            e.add_child(Node.u16('gp', lobby.get_int('gp')))
+            e.add_child(Node.u8_array('la', lobby.get_int_array('la', 4)))
+
+        return root
+
+    def handle_lobby_read_request(self, request: Node) -> Node:
+        root = Node.void('lobby')
+        root.add_child(Node.s32('interval', 120))
+        root.add_child(Node.s32('interval_p', 120))
+
+        # Look up all lobbies matching the criteria specified
+        mg = request.child_value('m_grade')  # noqa: F841
+        extid = request.child_value('uid')
+        limit = request.child_value('max')
+        userid = self.data.remote.user.from_extid(self.game, self.version, extid)
+        if userid is not None:
+            lobbies = self.data.local.lobby.get_all_lobbies(self.game, self.version)
+            for (user, lobby) in lobbies:
+                if limit <= 0:
+                    break
+
+                if user == userid:
+                    # If we have our own lobby, don't return it
+                    continue
+
+                profile = self.get_profile(user)
+                if profile is None:
+                    # No profile info, don't return this lobby
+                    continue
+
+                e = Node.void('e')
+                root.add_child(e)
+                e.add_child(Node.s32('eid', lobby.get_int('id')))
+                e.add_child(Node.u16('mid', lobby.get_int('mid')))
+                e.add_child(Node.u8('ng', lobby.get_int('ng')))
+                e.add_child(Node.s32('uid', profile.get_int('extid')))
+                e.add_child(Node.string('pn', profile.get_str('name')))
+                e.add_child(Node.s32('uattr', profile.get_int('uattr')))
+                e.add_child(Node.s32('mopt', lobby.get_int('mopt')))
+                e.add_child(Node.s16('mg', profile.get_int('mg')))
+                e.add_child(Node.s32('tid', lobby.get_int('tid')))
+                e.add_child(Node.string('tn', lobby.get_str('tn')))
+                e.add_child(Node.s32('topt', lobby.get_int('topt')))
+                e.add_child(Node.string('lid', lobby.get_str('lid')))
+                e.add_child(Node.string('sn', lobby.get_str('sn')))
+                e.add_child(Node.u8('pref', lobby.get_int('pref')))
+                e.add_child(Node.s8('stg', lobby.get_int('stg')))
+                e.add_child(Node.s8('pside', lobby.get_int('pside')))
+                e.add_child(Node.s16('eatime', lobby.get_int('eatime')))
+                e.add_child(Node.u8_array('ga', lobby.get_int_array('ga', 4)))
+                e.add_child(Node.u16('gp', lobby.get_int('gp')))
+                e.add_child(Node.u8_array('la', lobby.get_int_array('la', 4)))
+
+                limit = limit - 1
+
+        return root
+
+    def handle_lobby_delete_request(self, request: Node) -> Node:
+        eid = request.child_value('eid')
+        self.data.local.lobby.destroy_lobby(eid)
+        return Node.void('lobby')
+
+    def handle_player_start_request(self, request: Node) -> Node:
+        # Add a dummy entry into the lobby setup so we can clean up on end play
+        refid = request.child_value('rid')
+        userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        if userid is not None:
+            self.data.local.lobby.put_play_session_info(
+                self.game,
+                self.version,
+                userid,
+                {}
+            )
+
+        root = Node.void('player')
+        root.add_child(Node.bool('is_suc', True))
+
+        unlock_music = Node.void('unlock_music')
+        root.add_child(unlock_music)
+        unlock_item = Node.void('unlock_item')
+        root.add_child(unlock_item)
+        item_lock_ctrl = Node.void('item_lock_ctrl')
+        root.add_child(item_lock_ctrl)
+
+        lincle_link_4 = Node.void('lincle_link_4')
+        root.add_child(lincle_link_4)
+        lincle_link_4.add_child(Node.u32('qpro', 0))
+        lincle_link_4.add_child(Node.u32('glass', 0))
+        lincle_link_4.add_child(Node.u32('treasure', 0))
+        lincle_link_4.add_child(Node.bool('for_iidx_0_0', False))
+        lincle_link_4.add_child(Node.bool('for_iidx_0_1', False))
+        lincle_link_4.add_child(Node.bool('for_iidx_0_2', False))
+        lincle_link_4.add_child(Node.bool('for_iidx_0_3', False))
+        lincle_link_4.add_child(Node.bool('for_iidx_0_4', False))
+        lincle_link_4.add_child(Node.bool('for_iidx_0_5', False))
+        lincle_link_4.add_child(Node.bool('for_iidx_0_6', False))
+        lincle_link_4.add_child(Node.bool('for_iidx_0', False))
+        lincle_link_4.add_child(Node.bool('for_iidx_1', False))
+        lincle_link_4.add_child(Node.bool('for_iidx_2', False))
+        lincle_link_4.add_child(Node.bool('for_iidx_3', False))
+        lincle_link_4.add_child(Node.bool('for_iidx_4', False))
+        lincle_link_4.add_child(Node.bool('for_rb_0_0', False))
+        lincle_link_4.add_child(Node.bool('for_rb_0_1', False))
+        lincle_link_4.add_child(Node.bool('for_rb_0_2', False))
+        lincle_link_4.add_child(Node.bool('for_rb_0_3', False))
+        lincle_link_4.add_child(Node.bool('for_rb_0_4', False))
+        lincle_link_4.add_child(Node.bool('for_rb_0_5', False))
+        lincle_link_4.add_child(Node.bool('for_rb_0_6', False))
+        lincle_link_4.add_child(Node.bool('for_rb_0', False))
+        lincle_link_4.add_child(Node.bool('for_rb_1', False))
+        lincle_link_4.add_child(Node.bool('for_rb_2', False))
+        lincle_link_4.add_child(Node.bool('for_rb_3', False))
+        lincle_link_4.add_child(Node.bool('for_rb_4', False))
+        lincle_link_4.add_child(Node.bool('qproflg', False))
+        lincle_link_4.add_child(Node.bool('glassflg', False))
+        lincle_link_4.add_child(Node.bool('complete', False))
+
+        # Add event info
+        self.__add_event_info(root)
+
+        return root
+
+    def handle_player_delete_request(self, request: Node) -> Node:
+        return Node.void('player')
+
+    def handle_player_end_request(self, request: Node) -> Node:
+        # Destroy play session based on info from the request
+        refid = request.child_value('rid')
+        userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        if userid is not None:
+            # Kill any lingering lobbies by this user
+            lobby = self.data.local.lobby.get_lobby(
+                self.game,
+                self.version,
+                userid,
+            )
+            if lobby is not None:
+                self.data.local.lobby.destroy_lobby(lobby.get_int('id'))
+            self.data.local.lobby.destroy_play_session_info(self.game, self.version, userid)
+
+        return Node.void('player')
+
+    def handle_player_read_request(self, request: Node) -> Node:
+        refid = request.child_value('rid')
+        profile = self.get_profile_by_refid(refid)
+        if profile:
+            return profile
+        return Node.void('player')
+
+    def handle_player_write_request(self, request: Node) -> Node:
+        refid = request.child_value('rid')
+        profile = self.put_profile_by_refid(refid, request)
+        root = Node.void('player')
+
+        if profile is None:
+            root.add_child(Node.s32('uid', 0))
+        else:
+            root.add_child(Node.s32('uid', profile.get_int('extid')))
+        root.add_child(Node.s32('time', Time.now()))
+        return root
+
+    def format_profile(self, userid: UserID, profile: ValidatedDict) -> Node:
+        statistics = self.get_play_statistics(userid)
+        game_config = self.get_game_config()
+        achievements = self.data.local.user.get_achievements(self.game, self.version, userid)
+        scores = self.data.remote.music.get_scores(self.game, self.version, userid)
+        links = self.data.local.user.get_links(self.game, self.version, userid)
+        root = Node.void('player')
+        pdata = Node.void('pdata')
+        root.add_child(pdata)
+
+        base = Node.void('base')
+        pdata.add_child(base)
+        base.add_child(Node.s32('uid', profile.get_int('extid')))
+        base.add_child(Node.string('name', profile.get_str('name')))
+        base.add_child(Node.s16('icon_id', profile.get_int('icon')))
+        base.add_child(Node.s16('lv', profile.get_int('lvl')))
+        base.add_child(Node.s32('exp', profile.get_int('exp')))
+        base.add_child(Node.s16('mg', profile.get_int('mg')))
+        base.add_child(Node.s16('ap', profile.get_int('ap')))
+        base.add_child(Node.s32('pc', profile.get_int('pc')))
+        base.add_child(Node.s32('uattr', profile.get_int('uattr')))
+
+        last_play_date = statistics.get_int_array('last_play_date', 3)
+        today_play_date = Time.todays_date()
+        if (
+            last_play_date[0] == today_play_date[0] and
+            last_play_date[1] == today_play_date[1] and
+            last_play_date[2] == today_play_date[2]
+        ):
+            today_count = statistics.get_int('today_plays', 0)
+        else:
+            today_count = 0
+
+        con = Node.void('con')
+        pdata.add_child(con)
+        con.add_child(Node.s32('day', today_count))
+        con.add_child(Node.s32('cnt', statistics.get_int('total_plays')))
+        con.add_child(Node.s32('total_cnt', statistics.get_int('total_plays')))
+        con.add_child(Node.s32('last', statistics.get_int('last_play_timestamp')))
+        con.add_child(Node.s32('now', Time.now()))
+
+        team = Node.void('team')
+        pdata.add_child(team)
+        team.add_child(Node.s32('id', profile.get_int('team_id', -1)))
+        team.add_child(Node.string('name', profile.get_str('team_name', '')))
+
+        custom = Node.void('custom')
+        customdict = profile.get_dict('custom')
+        pdata.add_child(custom)
+        custom.add_child(Node.u8('s_gls', customdict.get_int('s_gls')))
+        custom.add_child(Node.u8('bgm_m', customdict.get_int('bgm_m')))
+        custom.add_child(Node.u8('st_f', customdict.get_int('st_f')))
+        custom.add_child(Node.u8('st_bg', customdict.get_int('st_bg')))
+        custom.add_child(Node.u8('st_bg_b', customdict.get_int('st_bg_b')))
+        custom.add_child(Node.u8('eff_e', customdict.get_int('eff_e')))
+        custom.add_child(Node.u8('se_s', customdict.get_int('se_s')))
+        custom.add_child(Node.u8('se_s_v', customdict.get_int('se_s_v')))
+        custom.add_child(Node.s16('last_music_id', customdict.get_int('last_music_id')))
+        custom.add_child(Node.u8('last_note_grade', customdict.get_int('last_note_grade')))
+        custom.add_child(Node.u8('sort_type', customdict.get_int('sort_type')))
+        custom.add_child(Node.u8('narrowdown_type', customdict.get_int('narrowdown_type')))
+        custom.add_child(Node.bool('is_begginer', customdict.get_bool('is_begginer')))  # Yes, this is spelled right
+        custom.add_child(Node.bool('is_tut', customdict.get_bool('is_tut')))
+        custom.add_child(Node.s16_array('symbol_chat_0', customdict.get_int_array('symbol_chat_0', 6)))
+        custom.add_child(Node.s16_array('symbol_chat_1', customdict.get_int_array('symbol_chat_1', 6)))
+        custom.add_child(Node.u8('gauge_style', customdict.get_int('gauge_style')))
+        custom.add_child(Node.u8('obj_shade', customdict.get_int('obj_shade')))
+        custom.add_child(Node.u8('obj_size', customdict.get_int('obj_size')))
+        custom.add_child(Node.s16_array('byword', customdict.get_int_array('byword', 2)))
+        custom.add_child(Node.bool_array('is_auto_byword', customdict.get_bool_array('is_auto_byword', 2)))
+        custom.add_child(Node.bool('is_tweet', customdict.get_bool('is_tweet')))
+        custom.add_child(Node.bool('is_link_twitter', customdict.get_bool('is_link_twitter')))
+        custom.add_child(Node.s16('mrec_type', customdict.get_int('mrec_type')))
+        custom.add_child(Node.s16('card_disp_type', customdict.get_int('card_disp_type')))
+        custom.add_child(Node.s16('tab_sel', customdict.get_int('tab_sel')))
+        custom.add_child(Node.s32_array('hidden_param', customdict.get_int_array('hidden_param', 20)))
+
+        released = Node.void('released')
+        pdata.add_child(released)
+
+        for item in achievements:
+            if item.type[:5] != 'item_':
+                continue
+            itemtype = int(item.type[5:])
+            if game_config.get_bool('force_unlock_songs') and itemtype == 0:
+                # Don't echo unlocks when we're force unlocking, we'll do it later
+                continue
+
+            info = Node.void('info')
+            released.add_child(info)
+            info.add_child(Node.u8('type', itemtype))
+            info.add_child(Node.u16('id', item.id))
+            info.add_child(Node.u16('param', item.data.get_int('param')))
+
+        if game_config.get_bool('force_unlock_songs'):
+            ids: Dict[int, int] = {}
+            songs = self.data.local.music.get_all_songs(self.game, self.version)
+            for song in songs:
+                if song.id not in ids:
+                    ids[song.id] = 0
+
+                if song.data.get_int('difficulty') > 0:
+                    ids[song.id] = ids[song.id] | (1 << song.chart)
+
+            for songid in ids:
+                if ids[songid] == 0:
+                    continue
+
+                info = Node.void('info')
+                released.add_child(info)
+                info.add_child(Node.u8('type', 0))
+                info.add_child(Node.u16('id', songid))
+                info.add_child(Node.u16('param', ids[songid]))
+
+        # Scores
+        record = Node.void('record')
+        pdata.add_child(record)
+
+        for score in scores:
+            rec = Node.void('rec')
+            record.add_child(rec)
+            rec.add_child(Node.u16('mid', score.id))
+            rec.add_child(Node.u8('ng', score.chart))
+            rec.add_child(Node.s32('point', score.data.get_dict('stats').get_int('earned_points')))
+            rec.add_child(Node.s32('played_time', score.timestamp))
+
+            mrec_0 = Node.void('mrec_0')
+            rec.add_child(mrec_0)
+            mrec_0.add_child(Node.s32('win', score.data.get_dict('stats').get_int('win')))
+            mrec_0.add_child(Node.s32('lose', score.data.get_dict('stats').get_int('lose')))
+            mrec_0.add_child(Node.s32('draw', score.data.get_dict('stats').get_int('draw')))
+            mrec_0.add_child(Node.u8('ct', self.__db_to_game_clear_type(score.data.get_int('clear_type'), score.data.get_int('combo_type'))))
+            mrec_0.add_child(Node.s16('ar', int(score.data.get_int('achievement_rate') / 10)))
+            mrec_0.add_child(Node.s32('bs', score.points))
+            mrec_0.add_child(Node.s16('mc', score.data.get_int('combo')))
+            mrec_0.add_child(Node.s16('bmc', score.data.get_int('miss_count')))
+
+            mrec_1 = Node.void('mrec_1')
+            rec.add_child(mrec_1)
+            mrec_1.add_child(Node.s32('win', 0))
+            mrec_1.add_child(Node.s32('lose', 0))
+            mrec_1.add_child(Node.s32('draw', 0))
+            mrec_1.add_child(Node.u8('ct', 0))
+            mrec_1.add_child(Node.s16('ar', 0))
+            mrec_1.add_child(Node.s32('bs', 0))
+            mrec_1.add_child(Node.s16('mc', 0))
+            mrec_1.add_child(Node.s16('bmc', -1))
+
+        # Comment (seems unused?)
+        pdata.add_child(Node.string('cmnt', ''))
+
+        # Rivals
+        rival = Node.void('rival')
+        pdata.add_child(rival)
+
+        slotid = 0
+        for link in links:
+            if link.type != 'rival':
+                continue
+
+            rprofile = self.get_profile(link.other_userid)
+            if rprofile is None:
+                continue
+
+            r = Node.void('r')
+            rival.add_child(r)
+            r.add_child(Node.u8('slot_id', slotid))
+            r.add_child(Node.string('name', rprofile.get_str('name')))
+            r.add_child(Node.s32('id', rprofile.get_int('extid')))
+            r.add_child(Node.bool('friend', True))
+            r.add_child(Node.bool('locked', False))
+            r.add_child(Node.s32('rc', 0))
+            slotid = slotid + 1
+
+        # Glass points
+        glass = Node.void('glass')
+        pdata.add_child(glass)
+
+        for item in achievements:
+            if item.type != 'glass':
+                continue
+
+            g = Node.void('g')
+            glass.add_child(g)
+            g.add_child(Node.s32('id', item.id))
+            g.add_child(Node.s32('exp', item.data.get_int('exp')))
+
+        # Favorite music
+        fav_music_slot = Node.void('fav_music_slot')
+        pdata.add_child(fav_music_slot)
+
+        for item in achievements:
+            if item.type != 'music':
+                continue
+
+            slot = Node.void('slot')
+            fav_music_slot.add_child(slot)
+            slot.add_child(Node.u8('slot_id', item.id))
+            slot.add_child(Node.s16('music_id', item.data.get_int('music_id')))
+
+        narrow_down = Node.void('narrow_down')
+        pdata.add_child(narrow_down)
+        narrow_down.add_child(Node.s32_array('adv_param', [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]))
+
+        return root
+
+    def unformat_profile(self, userid: UserID, request: Node, oldprofile: ValidatedDict) -> ValidatedDict:
+        game_config = self.get_game_config()
+        newprofile = copy.deepcopy(oldprofile)
+
+        newprofile.replace_int('lid', ID.parse_machine_id(request.child_value('lid')))
+        newprofile.replace_str('name', request.child_value('pdata/base/name'))
+        newprofile.replace_int('icon', request.child_value('pdata/base/icon_id'))
+        newprofile.replace_int('lvl', request.child_value('pdata/base/lv'))
+        newprofile.replace_int('exp', request.child_value('pdata/base/exp'))
+        newprofile.replace_int('mg', request.child_value('pdata/base/mg'))
+        newprofile.replace_int('ap', request.child_value('pdata/base/ap'))
+        newprofile.replace_int('pc', request.child_value('pdata/base/pc'))
+        newprofile.replace_int('uattr', request.child_value('pdata/base/uattr'))
+
+        customdict = newprofile.get_dict('custom')
+        custom = request.child('pdata/custom')
+        if custom:
+            customdict.replace_int('s_gls', custom.child_value('s_gls'))
+            customdict.replace_int('bgm_m', custom.child_value('bgm_m'))
+            customdict.replace_int('st_f', custom.child_value('st_f'))
+            customdict.replace_int('st_bg', custom.child_value('st_bg'))
+            customdict.replace_int('st_bg_b', custom.child_value('st_bg_b'))
+            customdict.replace_int('eff_e', custom.child_value('eff_e'))
+            customdict.replace_int('se_s', custom.child_value('se_s'))
+            customdict.replace_int('se_s_v', custom.child_value('se_s_v'))
+            customdict.replace_int('last_music_id', custom.child_value('last_music_id'))
+            customdict.replace_int('last_note_grade', custom.child_value('last_note_grade'))
+            customdict.replace_int('sort_type', custom.child_value('sort_type'))
+            customdict.replace_int('narrowdown_type', custom.child_value('narrowdown_type'))
+            customdict.replace_bool('is_begginer', custom.child_value('is_begginer'))  # Yes, this is spelled right
+            customdict.replace_bool('is_tut', custom.child_value('is_tut'))
+            customdict.replace_int_array('symbol_chat_0', 6, custom.child_value('symbol_chat_0'))
+            customdict.replace_int_array('symbol_chat_1', 6, custom.child_value('symbol_chat_1'))
+            customdict.replace_int('gauge_style', custom.child_value('gauge_style'))
+            customdict.replace_int('obj_shade', custom.child_value('obj_shade'))
+            customdict.replace_int('obj_size', custom.child_value('obj_size'))
+            customdict.replace_int_array('byword', 2, custom.child_value('byword'))
+            customdict.replace_bool_array('is_auto_byword', 2, custom.child_value('is_auto_byword'))
+            customdict.replace_bool('is_tweet', custom.child_value('is_tweet'))
+            customdict.replace_bool('is_link_twitter', custom.child_value('is_link_twitter'))
+            customdict.replace_int('mrec_type', custom.child_value('mrec_type'))
+            customdict.replace_int('card_disp_type', custom.child_value('card_disp_type'))
+            customdict.replace_int('tab_sel', custom.child_value('tab_sel'))
+            customdict.replace_int_array('hidden_param', 20, custom.child_value('hidden_param'))
+        newprofile.replace_dict('custom', customdict)
+
+        # Music unlocks and other stuff
+        released = request.child('pdata/released')
+        if released:
+            for child in released.children:
+                if child.name != 'info':
+                    continue
+
+                item_id = child.child_value('id')
+                item_type = child.child_value('type')
+                param = child.child_value('param')
+                if game_config.get_bool('force_unlock_songs') and item_type == 0:
+                    # Don't save unlocks when we're force unlocking
+                    continue
+
+                self.data.local.user.put_achievement(
+                    self.game,
+                    self.version,
+                    userid,
+                    item_id,
+                    'item_{}'.format(item_type),
+                    {
+                        'param': param,
+                    },
+                )
+
+        # Grab any new records set during this play session. Reflec Beat Limelight only sends
+        # the top record back for songs that were played at least once during the session.
+        # Note that it sends the top record, so if you play the song twice, it will return
+        # only one record. Also, if you get a lower score than a previous try, it will return
+        # the previous try. So, we must also look at the battle log for the actual play scores,
+        # and combine the data if we can.
+        savedrecords: Dict[int, Dict[int, Dict[str, int]]] = {}
+        songplays = request.child('pdata/record')
+        if songplays:
+            for child in songplays.children:
+                if child.name != 'rec':
+                    continue
+
+                songid = child.child_value('mid')
+                chart = child.child_value('ng')
+
+                # These don't get sent with the battle logs, so we try to construct
+                # the values here.
+                if songid not in savedrecords:
+                    savedrecords[songid] = {}
+                savedrecords[songid][chart] = {
+                    'achievement_rate': child.child_value('mrec_0/ar') * 10,
+                    'points': child.child_value('mrec_0/bs'),
+                    'combo': child.child_value('mrec_0/mc'),
+                    'miss_count': child.child_value('mrec_0/bmc'),
+                    'win': child.child_value('mrec_0/win'),
+                    'lose': child.child_value('mrec_0/lose'),
+                    'draw': child.child_value('mrec_0/draw'),
+                    'earned_points': child.child_value('point'),
+                }
+
+        # Now, see the actual battles that were played. If we can, unify the data with a record.
+        # We only do that when the record achievement rate and score matches the battle achievement
+        # rate and score, so we know for a fact that that record was generated by this battle.
+        battlelogs = request.child('pdata/blog')
+        if battlelogs:
+            for child in battlelogs.children:
+                if child.name != 'log':
+                    continue
+
+                songid = child.child_value('mid')
+                chart = child.child_value('ng')
+
+                clear_type = child.child_value('myself/ct')
+                achievement_rate = child.child_value('myself/ar') * 10
+                points = child.child_value('myself/s')
+
+                clear_type, combo_type = self.__game_to_db_clear_type(clear_type)
+
+                combo = None
+                miss_count = -1
+                stats = None
+
+                if songid in savedrecords:
+                    if chart in savedrecords[songid]:
+                        data = savedrecords[songid][chart]
+
+                        if (
+                            data['achievement_rate'] == achievement_rate and
+                            data['points'] == points
+                        ):
+                            # This is the same record! Use the stats from it to update our
+                            # internal representation.
+                            combo = data['combo']
+                            miss_count = data['miss_count']
+                            stats = {
+                                'win': data['win'],
+                                'lose': data['lose'],
+                                'draw': data['draw'],
+                                'earned_points': data['earned_points'],
+                            }
+
+                self.update_score(
+                    userid,
+                    songid,
+                    chart,
+                    points,
+                    achievement_rate,
+                    clear_type,
+                    combo_type,
+                    miss_count,
+                    combo=combo,
+                    stats=stats,
+                )
+
+        # Keep track of glass points so unlocks work
+        glass = request.child('pdata/glass')
+        if glass:
+            for child in glass.children:
+                if child.name != 'g':
+                    continue
+
+                gid = child.child_value('id')
+                exp = child.child_value('exp')
+                self.data.local.user.put_achievement(
+                    self.game,
+                    self.version,
+                    userid,
+                    gid,
+                    'glass',
+                    {
+                        'exp': exp,
+                    },
+                )
+
+        # Keep track of favorite music selections
+        fav_music_slot = request.child('pdata/fav_music_slot')
+        if fav_music_slot:
+            for child in fav_music_slot.children:
+                if child.name != 'slot':
+                    continue
+
+                slot_id = child.child_value('slot_id')
+                music_id = child.child_value('music_id')
+                if music_id == -1:
+                    # Delete this favorite
+                    self.data.local.user.destroy_achievement(
+                        self.game,
+                        self.version,
+                        userid,
+                        slot_id,
+                        'music',
+                    )
+                else:
+                    # Add/update this favorite
+                    self.data.local.user.put_achievement(
+                        self.game,
+                        self.version,
+                        userid,
+                        slot_id,
+                        'music',
+                        {
+                            'music_id': music_id,
+                        },
+                    )
+
+        # Keep track of play statistics
+        self.update_play_statistics(userid)
+
+        return newprofile

bemani/backend/reflec/reflecbeat.py

@@ -0,0 +1,543 @@
+import copy
+from typing import Any, Dict, Tuple
+
+from bemani.backend.reflec.base import ReflecBeatBase
+
+from bemani.common import ValidatedDict, VersionConstants, ID, Time
+from bemani.data import UserID
+from bemani.protocol import Node
+
+
+class ReflecBeat(ReflecBeatBase):
+
+    name = "REFLEC BEAT"
+    version = VersionConstants.REFLEC_BEAT
+
+    # Clear types according to the game
+    GAME_CLEAR_TYPE_NO_PLAY = 0
+    GAME_CLEAR_TYPE_PLAYED = 2
+    GAME_CLEAR_TYPE_FULL_COMBO = 3
+
+    @classmethod
+    def get_settings(cls) -> Dict[str, Any]:
+        """
+        Return all of our front-end modifiably settings.
+        """
+        return {
+            'bools': [
+                {
+                    'name': 'Force Song Unlock',
+                    'tip': 'Force unlock all songs.',
+                    'category': 'game_config',
+                    'setting': 'force_unlock_songs',
+                },
+            ],
+            'ints': [],
+        }
+
+    def __db_to_game_clear_type(self, db_clear_type: int, db_combo_type: int) -> int:
+        if db_clear_type == self.CLEAR_TYPE_NO_PLAY:
+            return self.GAME_CLEAR_TYPE_NO_PLAY
+        if db_clear_type == self.CLEAR_TYPE_FAILED:
+            return self.GAME_CLEAR_TYPE_PLAYED
+        if db_clear_type in [
+            self.CLEAR_TYPE_CLEARED,
+            self.CLEAR_TYPE_HARD_CLEARED,
+            self.CLEAR_TYPE_S_HARD_CLEARED,
+        ]:
+            if db_combo_type in [
+                self.COMBO_TYPE_NONE,
+                self.COMBO_TYPE_ALMOST_COMBO,
+            ]:
+                return self.GAME_CLEAR_TYPE_PLAYED
+            if db_combo_type in [
+                self.COMBO_TYPE_FULL_COMBO,
+                self.COMBO_TYPE_FULL_COMBO_ALL_JUST,
+            ]:
+                return self.GAME_CLEAR_TYPE_FULL_COMBO
+
+            raise Exception('Invalid db_combo_type {}'.format(db_combo_type))
+        raise Exception('Invalid db_clear_type {}'.format(db_clear_type))
+
+    def __game_to_db_clear_type(self, game_clear_type: int, game_achievement_rate: int) -> Tuple[int, int]:
+        if game_clear_type == self.GAME_CLEAR_TYPE_NO_PLAY:
+            return (self.CLEAR_TYPE_NO_PLAY, self.COMBO_TYPE_NONE)
+        if game_clear_type == self.GAME_CLEAR_TYPE_PLAYED:
+            if game_achievement_rate >= 7000:
+                return (self.CLEAR_TYPE_CLEARED, self.COMBO_TYPE_NONE)
+            else:
+                return (self.CLEAR_TYPE_FAILED, self.COMBO_TYPE_NONE)
+        if game_clear_type == self.GAME_CLEAR_TYPE_FULL_COMBO:
+            return (self.CLEAR_TYPE_CLEARED, self.COMBO_TYPE_FULL_COMBO)
+
+        raise Exception('Invalid game_clear_type {}'.format(game_clear_type))
+
+    def handle_log_pcb_status_request(self, request: Node) -> Node:
+        return Node.void('log')
+
+    def handle_log_opsetting_request(self, request: Node) -> Node:
+        return Node.void('log')
+
+    def handle_log_play_request(self, request: Node) -> Node:
+        return Node.void('log')
+
+    def handle_pcbinfo_get_request(self, request: Node) -> Node:
+        shop_id = ID.parse_machine_id(request.child_value('lid'))
+        machine = self.get_machine_by_id(shop_id)
+        if machine is not None:
+            machine_name = machine.name
+            close = machine.data.get_bool('close')
+            hour = machine.data.get_int('hour')
+            minute = machine.data.get_int('minute')
+            pref = machine.data.get_int('pref', 51)
+        else:
+            machine_name = ''
+            close = False
+            hour = 0
+            minute = 0
+            pref = 51
+
+        root = Node.void('pcbinfo')
+        info = Node.void('info')
+        root.add_child(info)
+
+        info.add_child(Node.string('name', machine_name))
+        info.add_child(Node.s16('pref', pref))
+        info.add_child(Node.bool('close', close))
+        info.add_child(Node.u8('hour', hour))
+        info.add_child(Node.u8('min', minute))
+
+        return root
+
+    def handle_pcbinfo_set_request(self, request: Node) -> Node:
+        self.update_machine_name(request.child_value('info/name'))
+        self.update_machine_data({
+            'close': request.child_value('info/close'),
+            'hour': request.child_value('info/hour'),
+            'minute': request.child_value('info/min'),
+            'pref': request.child_value('info/pref'),
+        })
+        return Node.void('pcbinfo')
+
+    def __add_event_info(self, request: Node) -> None:
+        events: Dict[int, int] = {}
+
+        for (eventid, phase) in events.items():
+            data = Node.void('data')
+            request.add_child(data)
+            data.add_child(Node.s32('type', -1))
+            data.add_child(Node.s32('value', -1))
+
+    def handle_sysinfo_get_request(self, request: Node) -> Node:
+        root = Node.void('sysinfo')
+        trd = Node.void('trd')
+        root.add_child(trd)
+
+        # Add event info
+        self.__add_event_info(trd)
+
+        return root
+
+    def handle_sysinfo_fan_request(self, request: Node) -> Node:
+        sysinfo = Node.void('sysinfo')
+        sysinfo.add_child(Node.u8('pref', 51))
+        sysinfo.add_child(Node.string('lid', request.child_value('lid')))
+        return sysinfo
+
+    def handle_lobby_entry_request(self, request: Node) -> Node:
+        root = Node.void('lobby')
+
+        # Create a lobby entry for this user
+        extid = request.child_value('e/uid')
+        userid = self.data.remote.user.from_extid(self.game, self.version, extid)
+        if userid is not None:
+            profile = self.get_profile(userid)
+            self.data.local.lobby.put_lobby(
+                self.game,
+                self.version,
+                userid,
+                {
+                    'mid': request.child_value('e/mid'),
+                    'ng': request.child_value('e/ng'),
+                    'lid': request.child_value('e/lid'),
+                    'sn': request.child_value('e/sn'),
+                    'pref': request.child_value('e/pref'),
+                    'ga': request.child_value('e/ga'),
+                    'gp': request.child_value('e/gp'),
+                    'la': request.child_value('e/la'),
+                }
+            )
+            lobby = self.data.local.lobby.get_lobby(
+                self.game,
+                self.version,
+                userid,
+            )
+            root.add_child(Node.s32('eid', lobby.get_int('id')))
+            e = Node.void('e')
+            root.add_child(e)
+            e.add_child(Node.s32('eid', lobby.get_int('id')))
+            e.add_child(Node.u16('mid', lobby.get_int('mid')))
+            e.add_child(Node.u8('ng', lobby.get_int('ng')))
+            e.add_child(Node.s32('uid', profile.get_int('extid')))
+            e.add_child(Node.string('pn', profile.get_str('name')))
+            e.add_child(Node.s32('exp', profile.get_int('exp')))
+            e.add_child(Node.u8('mg', profile.get_int('mg')))
+            e.add_child(Node.s32('tid', lobby.get_int('tid')))
+            e.add_child(Node.string('tn', lobby.get_str('tn')))
+            e.add_child(Node.string('lid', lobby.get_str('lid')))
+            e.add_child(Node.string('sn', lobby.get_str('sn')))
+            e.add_child(Node.u8('pref', lobby.get_int('pref')))
+            e.add_child(Node.u8_array('ga', lobby.get_int_array('ga', 4)))
+            e.add_child(Node.u16('gp', lobby.get_int('gp')))
+            e.add_child(Node.u8_array('la', lobby.get_int_array('la', 4)))
+
+        return root
+
+    def handle_lobby_read_request(self, request: Node) -> Node:
+        root = Node.void('lobby')
+
+        # Look up all lobbies matching the criteria specified
+        mg = request.child_value('m_grade')  # noqa: F841
+        extid = request.child_value('uid')
+        limit = request.child_value('max')
+        userid = self.data.remote.user.from_extid(self.game, self.version, extid)
+        if userid is not None:
+            lobbies = self.data.local.lobby.get_all_lobbies(self.game, self.version)
+            for (user, lobby) in lobbies:
+                if limit <= 0:
+                    break
+
+                if user == userid:
+                    # If we have our own lobby, don't return it
+                    continue
+
+                profile = self.get_profile(user)
+                if profile is None:
+                    # No profile info, don't return this lobby
+                    continue
+
+                e = Node.void('e')
+                root.add_child(e)
+                e.add_child(Node.s32('eid', lobby.get_int('id')))
+                e.add_child(Node.u16('mid', lobby.get_int('mid')))
+                e.add_child(Node.u8('ng', lobby.get_int('ng')))
+                e.add_child(Node.s32('uid', profile.get_int('extid')))
+                e.add_child(Node.string('pn', profile.get_str('name')))
+                e.add_child(Node.s32('exp', profile.get_int('exp')))
+                e.add_child(Node.u8('mg', profile.get_int('mg')))
+                e.add_child(Node.s32('tid', lobby.get_int('tid')))
+                e.add_child(Node.string('tn', lobby.get_str('tn')))
+                e.add_child(Node.string('lid', lobby.get_str('lid')))
+                e.add_child(Node.string('sn', lobby.get_str('sn')))
+                e.add_child(Node.u8('pref', lobby.get_int('pref')))
+                e.add_child(Node.u8_array('ga', lobby.get_int_array('ga', 4)))
+                e.add_child(Node.u16('gp', lobby.get_int('gp')))
+                e.add_child(Node.u8_array('la', lobby.get_int_array('la', 4)))
+
+                limit = limit - 1
+
+        return root
+
+    def handle_lobby_delete_request(self, request: Node) -> Node:
+        eid = request.child_value('eid')
+        self.data.local.lobby.destroy_lobby(eid)
+        return Node.void('lobby')
+
+    def handle_player_start_request(self, request: Node) -> Node:
+        # Add a dummy entry into the lobby setup so we can clean up on end play
+        refid = request.child_value('rid')
+        userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        if userid is not None:
+            self.data.local.lobby.put_play_session_info(
+                self.game,
+                self.version,
+                userid,
+                {}
+            )
+
+        root = Node.void('player')
+        root.add_child(Node.bool('is_suc', True))
+
+        # Add event info
+        self.__add_event_info(root)
+
+        return root
+
+    def handle_player_delete_request(self, request: Node) -> Node:
+        return Node.void('player')
+
+    def handle_player_end_request(self, request: Node) -> Node:
+        # Destroy play session based on info from the request
+        refid = request.child_value('rid')
+        userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        if userid is not None:
+            # Kill any lingering lobbies by this user
+            lobby = self.data.local.lobby.get_lobby(
+                self.game,
+                self.version,
+                userid,
+            )
+            if lobby is not None:
+                self.data.local.lobby.destroy_lobby(lobby.get_int('id'))
+            self.data.local.lobby.destroy_play_session_info(self.game, self.version, userid)
+
+        return Node.void('player')
+
+    def handle_player_read_request(self, request: Node) -> Node:
+        refid = request.child_value('rid')
+        profile = self.get_profile_by_refid(refid)
+        if profile:
+            return profile
+        return Node.void('player')
+
+    def handle_player_write_request(self, request: Node) -> Node:
+        refid = request.child_value('rid')
+        profile = self.put_profile_by_refid(refid, request)
+        root = Node.void('player')
+
+        if profile is None:
+            root.add_child(Node.s32('uid', 0))
+        else:
+            root.add_child(Node.s32('uid', profile.get_int('extid')))
+        root.add_child(Node.s32('time', Time.now()))
+        return root
+
+    def format_profile(self, userid: UserID, profile: ValidatedDict) -> Node:
+        statistics = self.get_play_statistics(userid)
+        game_config = self.get_game_config()
+        achievements = self.data.local.user.get_achievements(self.game, self.version, userid)
+        scores = self.data.remote.music.get_scores(self.game, self.version, userid)
+        root = Node.void('player')
+        pdata = Node.void('pdata')
+        root.add_child(pdata)
+
+        base = Node.void('base')
+        pdata.add_child(base)
+        base.add_child(Node.s32('uid', profile.get_int('extid')))
+        base.add_child(Node.string('name', profile.get_str('name')))
+        base.add_child(Node.s16('lv', profile.get_int('lvl')))
+        base.add_child(Node.s32('exp', profile.get_int('exp')))
+        base.add_child(Node.s16('mg', profile.get_int('mg')))
+        base.add_child(Node.s16('ap', profile.get_int('ap')))
+        base.add_child(Node.s32('flag', profile.get_int('flag')))
+
+        last_play_date = statistics.get_int_array('last_play_date', 3)
+        today_play_date = Time.todays_date()
+        if (
+            last_play_date[0] == today_play_date[0] and
+            last_play_date[1] == today_play_date[1] and
+            last_play_date[2] == today_play_date[2]
+        ):
+            today_count = statistics.get_int('today_plays', 0)
+        else:
+            today_count = 0
+
+        con = Node.void('con')
+        pdata.add_child(con)
+        con.add_child(Node.s32('day', today_count))
+        con.add_child(Node.s32('cnt', statistics.get_int('total_plays')))
+        con.add_child(Node.s32('last', statistics.get_int('last_play_timestamp')))
+        con.add_child(Node.s32('now', Time.now()))
+
+        team = Node.void('team')
+        pdata.add_child(team)
+        team.add_child(Node.s32('id', -1))
+        team.add_child(Node.string('name', ''))
+
+        custom = Node.void('custom')
+        customdict = profile.get_dict('custom')
+        pdata.add_child(custom)
+        custom.add_child(Node.u8('bgm_m', customdict.get_int('bgm_m')))
+        custom.add_child(Node.u8('st_f', customdict.get_int('st_f')))
+        custom.add_child(Node.u8('st_bg', customdict.get_int('st_bg')))
+        custom.add_child(Node.u8('st_bg_b', customdict.get_int('st_bg_b')))
+        custom.add_child(Node.u8('eff_e', customdict.get_int('eff_e')))
+        custom.add_child(Node.u8('se_s', customdict.get_int('se_s')))
+        custom.add_child(Node.u8('se_s_v', customdict.get_int('se_s_v')))
+
+        released = Node.void('released')
+        pdata.add_child(released)
+
+        for item in achievements:
+            if item.type[:5] != 'item_':
+                continue
+            itemtype = int(item.type[5:])
+            if game_config.get_bool('force_unlock_songs') and itemtype == 0:
+                # Don't echo unlocks when we're force unlocking, we'll do it later
+                continue
+
+            info = Node.void('info')
+            released.add_child(info)
+            info.add_child(Node.u8('type', itemtype))
+            info.add_child(Node.u16('id', item.id))
+
+        if game_config.get_bool('force_unlock_songs'):
+            songs = {song.id for song in self.data.local.music.get_all_songs(self.game, self.version)}
+
+            for songid in songs:
+                info = Node.void('info')
+                released.add_child(info)
+                info.add_child(Node.u8('type', 0))
+                info.add_child(Node.u16('id', songid))
+
+        # Scores
+        record = Node.void('record')
+        pdata.add_child(record)
+
+        for score in scores:
+            rec = Node.void('rec')
+            record.add_child(rec)
+            rec.add_child(Node.u16('mid', score.id))
+            rec.add_child(Node.u8('ng', score.chart))
+            rec.add_child(Node.s32('win', score.data.get_dict('stats').get_int('win')))
+            rec.add_child(Node.s32('lose', score.data.get_dict('stats').get_int('lose')))
+            rec.add_child(Node.s32('draw', score.data.get_dict('stats').get_int('draw')))
+            rec.add_child(Node.u8('ct', self.__db_to_game_clear_type(score.data.get_int('clear_type'), score.data.get_int('combo_type'))))
+            rec.add_child(Node.s16('ar', int(score.data.get_int('achievement_rate') / 10)))
+            rec.add_child(Node.s16('bs', score.points))
+            rec.add_child(Node.s16('mc', score.data.get_int('combo')))
+            rec.add_child(Node.s16('bmc', score.data.get_int('miss_count')))
+
+        # In original ReflecBeat, the entire battle log was returned for each battle.
+        # We don't support storing all of that info, so don't return anything here.
+        blog = Node.void('blog')
+        pdata.add_child(blog)
+
+        # Comment (seems unused?)
+        pdata.add_child(Node.string('cmnt', ''))
+
+        return root
+
+    def unformat_profile(self, userid: UserID, request: Node, oldprofile: ValidatedDict) -> ValidatedDict:
+        game_config = self.get_game_config()
+        newprofile = copy.deepcopy(oldprofile)
+
+        newprofile.replace_int('lid', ID.parse_machine_id(request.child_value('lid')))
+        newprofile.replace_str('name', request.child_value('pdata/base/name'))
+        newprofile.replace_int('lvl', request.child_value('pdata/base/lv'))
+        newprofile.replace_int('exp', request.child_value('pdata/base/exp'))
+        newprofile.replace_int('mg', request.child_value('pdata/base/mg'))
+        newprofile.replace_int('ap', request.child_value('pdata/base/ap'))
+        newprofile.replace_int('flag', request.child_value('pdata/base/flag'))
+
+        customdict = newprofile.get_dict('custom')
+        custom = request.child('pdata/custom')
+        if custom:
+            customdict.replace_int('bgm_m', custom.child_value('bgm_m'))
+            customdict.replace_int('st_f', custom.child_value('st_f'))
+            customdict.replace_int('st_bg', custom.child_value('st_bg'))
+            customdict.replace_int('st_bg_b', custom.child_value('st_bg_b'))
+            customdict.replace_int('eff_e', custom.child_value('eff_e'))
+            customdict.replace_int('se_s', custom.child_value('se_s'))
+            customdict.replace_int('se_s_v', custom.child_value('se_s_v'))
+        newprofile.replace_dict('custom', customdict)
+
+        # Music unlocks and other stuff
+        released = request.child('pdata/released')
+        if released:
+            for child in released.children:
+                if child.name != 'info':
+                    continue
+
+                item_id = child.child_value('id')
+                item_type = child.child_value('type')
+                if game_config.get_bool('force_unlock_songs') and item_type == 0:
+                    # Don't save unlocks when we're force unlocking
+                    continue
+
+                self.data.local.user.put_achievement(
+                    self.game,
+                    self.version,
+                    userid,
+                    item_id,
+                    'item_{}'.format(item_type),
+                    {},
+                )
+
+        # Grab any new records set during this play session. Reflec Beat original only sends
+        # the top record back for songs that were played at least once during the session.
+        # Note that it sends the top record, so if you play the song twice, it will return
+        # only one record. Also, if you get a lower score than a previous try, it will return
+        # the previous try. So, we must also look at the battle log for the actual play scores,
+        # and combine the data if we can.
+        savedrecords: Dict[int, Dict[int, Dict[str, int]]] = {}
+        songplays = request.child('pdata/record')
+        if songplays:
+            for child in songplays.children:
+                if child.name != 'rec':
+                    continue
+
+                songid = child.child_value('mid')
+                chart = child.child_value('ng')
+
+                # These don't get sent with the battle logs, so we try to construct
+                # the values here.
+                if songid not in savedrecords:
+                    savedrecords[songid] = {}
+                savedrecords[songid][chart] = {
+                    'achievement_rate': child.child_value('ar') * 10,
+                    'points': child.child_value('bs'),
+                    'combo': child.child_value('mc'),
+                    'miss_count': child.child_value('bmc'),
+                    'win': child.child_value('win'),
+                    'lose': child.child_value('lose'),
+                    'draw': child.child_value('draw'),
+                }
+
+        # Now, see the actual battles that were played. If we can, unify the data with a record.
+        # We only do that when the record achievement rate and score matches the battle achievement
+        # rate and score, so we know for a fact that that record was generated by this battle.
+        battlelogs = request.child('pdata/blog')
+        if battlelogs:
+            for child in battlelogs.children:
+                if child.name != 'log':
+                    continue
+
+                songid = child.child_value('mid')
+                chart = child.child_value('ng')
+
+                clear_type = child.child_value('myself/ct')
+                achievement_rate = child.child_value('myself/ar') * 10
+                points = child.child_value('myself/s')
+
+                clear_type, combo_type = self.__game_to_db_clear_type(clear_type, achievement_rate)
+
+                combo = None
+                miss_count = -1
+                stats = None
+
+                if songid in savedrecords:
+                    if chart in savedrecords[songid]:
+                        data = savedrecords[songid][chart]
+
+                        if (
+                            data['achievement_rate'] == achievement_rate and
+                            data['points'] == points
+                        ):
+                            # This is the same record! Use the stats from it to update our
+                            # internal representation.
+                            combo = data['combo']
+                            miss_count = data['miss_count']
+                            stats = {
+                                'win': data['win'],
+                                'lose': data['lose'],
+                                'draw': data['draw'],
+                            }
+
+                self.update_score(
+                    userid,
+                    songid,
+                    chart,
+                    points,
+                    achievement_rate,
+                    clear_type,
+                    combo_type,
+                    miss_count,
+                    combo=combo,
+                    stats=stats,
+                )
+
+        # Keep track of play statistics
+        self.update_play_statistics(userid)
+
+        return newprofile

bemani/backend/reflec/volzza.py

@@ -0,0 +1,835 @@
+import copy
+from typing import Any, Dict, List, Optional
+
+from bemani.backend.reflec.base import ReflecBeatBase
+from bemani.backend.reflec.volzzabase import ReflecBeatVolzzaBase
+from bemani.backend.reflec.groovin import ReflecBeatGroovin
+
+from bemani.common import ValidatedDict, VersionConstants, ID, Time
+from bemani.data import Score, UserID
+from bemani.protocol import Node
+
+
+class ReflecBeatVolzza(ReflecBeatVolzzaBase):
+
+    name = "REFLEC BEAT VOLZZA"
+    version = VersionConstants.REFLEC_BEAT_VOLZZA
+
+    def previous_version(self) -> Optional[ReflecBeatBase]:
+        return ReflecBeatGroovin(self.data, self.config, self.model)
+
+    @classmethod
+    def get_settings(cls) -> Dict[str, Any]:
+        """
+        Return all of our front-end modifiably settings.
+        """
+        return {
+            'bools': [
+                {
+                    'name': 'Force Song Unlock',
+                    'tip': 'Force unlock all songs.',
+                    'category': 'game_config',
+                    'setting': 'force_unlock_songs',
+                },
+            ],
+            'ints': [],
+        }
+
+    def _add_event_info(self, root: Node) -> None:
+        event_ctrl = Node.void('event_ctrl')
+        root.add_child(event_ctrl)
+        # Contains zero or more nodes like:
+        # <data>
+        #     <type __type="s32">any</type>
+        #     <index __type="s32">any</phase>
+        #     <value __type="s32">any</phase>
+        #     <value2 __type="s32">any</phase>
+        #     <start_time __type="s32">any</phase>
+        #     <end_time __type="s32">any</phase>
+        # </data>
+
+        item_lock_ctrl = Node.void('item_lock_ctrl')
+        root.add_child(item_lock_ctrl)
+        # Contains zero or more nodes like:
+        # <item>
+        #     <type __type="u8">any</type>
+        #     <id __type="u16">any</id>
+        #     <param __type="u16">0-3</param>
+        # </item>
+
+    def handle_player_rb5_player_read_score_request(self, request: Node) -> Node:
+        refid = request.child_value('rid')
+        userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        if userid is None:
+            scores: List[Score] = []
+        else:
+            scores = self.data.remote.music.get_scores(self.game, self.version, userid)
+
+        root = Node.void('player')
+        pdata = Node.void('pdata')
+        root.add_child(pdata)
+
+        record = Node.void('record')
+        pdata.add_child(record)
+
+        for score in scores:
+            rec = Node.void('rec')
+            record.add_child(rec)
+            rec.add_child(Node.s16('mid', score.id))
+            rec.add_child(Node.s8('ntgrd', score.chart))
+            rec.add_child(Node.s32('pc', score.plays))
+            rec.add_child(Node.s8('ct', self._db_to_game_clear_type(score.data.get_int('clear_type'))))
+            rec.add_child(Node.s16('ar', score.data.get_int('achievement_rate')))
+            rec.add_child(Node.s16('scr', score.points))
+            rec.add_child(Node.s16('ms', score.data.get_int('miss_count')))
+            rec.add_child(Node.s16(
+                'param',
+                self._db_to_game_combo_type(score.data.get_int('combo_type')) + score.data.get_int('param'),
+            ))
+            rec.add_child(Node.s32('bscrt', score.timestamp))
+            rec.add_child(Node.s32('bart', score.data.get_int('best_achievement_rate_time')))
+            rec.add_child(Node.s32('bctt', score.data.get_int('best_clear_type_time')))
+            rec.add_child(Node.s32('bmst', score.data.get_int('best_miss_count_time')))
+            rec.add_child(Node.s32('time', score.data.get_int('last_played_time')))
+            rec.add_child(Node.s32('k_flag', score.data.get_int('kflag')))
+
+        return root
+
+    def handle_player_rb5_player_read_rival_score_request(self, request: Node) -> Node:
+        extid = request.child_value('uid')
+        songid = request.child_value('music_id')
+        chart = request.child_value('note_grade')
+        userid = self.data.remote.user.from_extid(self.game, self.version, extid)
+        if userid is None:
+            score = None
+            profile = None
+        else:
+            score = self.data.remote.music.get_score(self.game, self.version, userid, songid, chart)
+            profile = self.get_any_profile(userid)
+
+        root = Node.void('player')
+        if score is not None and profile is not None:
+            player_select_score = Node.void('player_select_score')
+            root.add_child(player_select_score)
+
+            player_select_score.add_child(Node.s32('user_id', extid))
+            player_select_score.add_child(Node.string('name', profile.get_str('name')))
+            player_select_score.add_child(Node.s32('m_score', score.points))
+            player_select_score.add_child(Node.s32('m_scoreTime', score.timestamp))
+            player_select_score.add_child(Node.s16('m_iconID', profile.get_dict('config').get_int('icon_id')))
+        return root
+
+    def handle_player_rb5_player_read_rival_ranking_data_request(self, request: Node) -> Node:
+        extid = request.child_value('uid')
+        userid = self.data.remote.user.from_extid(self.game, self.version, extid)
+
+        root = Node.void('player')
+        rival_data = Node.void('rival_data')
+        root.add_child(rival_data)
+
+        if userid is not None:
+            links = self.data.local.user.get_links(self.game, self.version, userid)
+            for link in links:
+                if link.type != 'rival':
+                    continue
+
+                rprofile = self.get_profile(link.other_userid)
+                if rprofile is None:
+                    continue
+
+                rl = Node.void('rl')
+                rival_data.add_child(rl)
+                rl.add_child(Node.s32('uid', rprofile.get_int('extid')))
+                rl.add_child(Node.string('nm', rprofile.get_str('name')))
+                rl.add_child(Node.s16('ic', rprofile.get_dict('config').get_int('icon_id')))
+
+                scores = self.data.remote.music.get_scores(self.game, self.version, link.other_userid)
+                scores_by_musicid: Dict[int, List[Score]] = {}
+                for score in scores:
+                    if score.id not in scores_by_musicid:
+                        scores_by_musicid[score.id] = [None, None, None, None]
+                    scores_by_musicid[score.id][score.chart] = score
+
+                for (mid, scores) in scores_by_musicid.items():
+                    points = [
+                        score.points << 32 if score is not None else 0
+                        for score in scores
+                    ]
+                    timestamps = [
+                        score.timestamp if score is not None else 0
+                        for score in scores
+                    ]
+
+                    sl = Node.void('sl')
+                    rl.add_child(sl)
+                    sl.add_child(Node.s16('mid', mid))
+                    # Score, but shifted left 32 bits for no reason
+                    sl.add_child(Node.u64_array('m', points))
+                    # Timestamp of the clear
+                    sl.add_child(Node.u64_array('t', timestamps))
+
+        return root
+
+    def handle_player_rb5_player_read_rank_request(self, request: Node) -> Node:
+        # This gives us a 6-integer array mapping to user scores for the following:
+        # [total score, basic chart score, medium chart score, hard chart score,
+        # special chart score]. It also returns the previous rank, but this is
+        # not used in-game as far as I can tell.
+        current_scores = request.child_value('sc')
+        current_minigame_score = request.child_value('mg_sc')
+
+        # First, grab all scores on the network for this version.
+        all_scores = self.data.remote.music.get_all_scores(self.game, self.version)
+
+        # Now grab all participating users that had scores
+        all_users = {userid for (userid, score) in all_scores}
+
+        # Now, group the scores by user, so we can add up the totals, only including
+        # scores where the user at least cleared the song.
+        scores_by_user = {
+            userid: [
+                score for (uid, score) in all_scores
+                if uid == userid and score.data.get_int('clear_type') >= self.CLEAR_TYPE_CLEARED]
+            for userid in all_users
+        }
+
+        # Now grab all user profiles for this game
+        all_profiles = {
+            profile[0]: profile[1] for profile in
+            self.data.remote.user.get_all_profiles(self.game, self.version)
+        }
+
+        # Now, sum up the scores into the five categories that the game expects.
+        total_scores = sorted(
+            [
+                sum([score.points for score in scores])
+                for userid, scores in scores_by_user.items()
+            ],
+            reverse=True,
+        )
+        basic_scores = sorted(
+            [
+                sum([score.points for score in scores if score.chart == self.CHART_TYPE_BASIC])
+                for userid, scores in scores_by_user.items()
+            ],
+            reverse=True,
+        )
+        medium_scores = sorted(
+            [
+                sum([score.points for score in scores if score.chart == self.CHART_TYPE_MEDIUM])
+                for userid, scores in scores_by_user.items()
+            ],
+            reverse=True,
+        )
+        hard_scores = sorted(
+            [
+                sum([score.points for score in scores if score.chart == self.CHART_TYPE_HARD])
+                for userid, scores in scores_by_user.items()
+            ],
+        )
+        special_scores = sorted(
+            [
+                sum([score.points for score in scores if score.chart == self.CHART_TYPE_SPECIAL])
+                for userid, scores in scores_by_user.items()
+            ],
+            reverse=True,
+        )
+        minigame_scores = sorted(
+            [
+                all_profiles.get(userid, ValidatedDict()).get_int('mgsc')
+                for userid in all_users
+            ],
+            reverse=True,
+        )
+
+        # Guarantee that a zero score is at the end of every list, so that it makes
+        # the algorithm for figuring out place have no edge case.
+        total_scores.append(0)
+        basic_scores.append(0)
+        medium_scores.append(0)
+        hard_scores.append(0)
+        special_scores.append(0)
+        minigame_scores.append(0)
+
+        # Now, figure out where we fit based on the scores sent from the game.
+        user_place = [1, 1, 1, 1, 1, 1]
+        which_score = [
+            total_scores,
+            basic_scores,
+            medium_scores,
+            hard_scores,
+            special_scores,
+            minigame_scores,
+        ]
+        earned_scores = current_scores + [current_minigame_score]
+        for i in range(len(user_place)):
+            earned_score = earned_scores[i]
+            scores = which_score[i]
+            for score in scores:
+                if earned_score >= score:
+                    break
+                user_place[i] = user_place[i] + 1
+
+        # Separate out minigame rank from scores
+        minigame_rank = user_place[-1]
+        user_place = user_place[:-1]
+
+        root = Node.void('player')
+
+        # Populate current ranking.
+        tbs = Node.void('tbs')
+        root.add_child(tbs)
+        tbs.add_child(Node.s32_array('new_rank', user_place))
+        tbs.add_child(Node.s32_array('old_rank', [-1, -1, -1, -1, -1]))
+
+        # Populate current minigame ranking (LOL).
+        mng = Node.void('mng')
+        root.add_child(mng)
+        mng.add_child(Node.s32('new_rank', minigame_rank))
+        mng.add_child(Node.s32('old_rank', -1))
+
+        return root
+
+    def handle_player_rb5_player_write_request(self, request: Node) -> Node:
+        refid = request.child_value('pdata/account/rid')
+        profile = self.put_profile_by_refid(refid, request)
+        root = Node.void('player')
+
+        if profile is None:
+            root.add_child(Node.s32('uid', 0))
+        else:
+            root.add_child(Node.s32('uid', profile.get_int('extid')))
+        return root
+
+    def format_profile(self, userid: UserID, profile: ValidatedDict) -> Node:
+        statistics = self.get_play_statistics(userid)
+        game_config = self.get_game_config()
+        achievements = self.data.local.user.get_achievements(self.game, self.version, userid)
+        links = self.data.local.user.get_links(self.game, self.version, userid)
+        root = Node.void('player')
+        pdata = Node.void('pdata')
+        root.add_child(pdata)
+
+        # Account time info
+        last_play_date = statistics.get_int_array('last_play_date', 3)
+        today_play_date = Time.todays_date()
+        if (
+            last_play_date[0] == today_play_date[0] and
+            last_play_date[1] == today_play_date[1] and
+            last_play_date[2] == today_play_date[2]
+        ):
+            today_count = statistics.get_int('today_plays', 0)
+        else:
+            today_count = 0
+
+        # Previous account info
+        previous_version = self.previous_version()
+        if previous_version:
+            succeeded = previous_version.has_profile(userid)
+        else:
+            succeeded = False
+
+        # Account info
+        account = Node.void('account')
+        pdata.add_child(account)
+        account.add_child(Node.s32('usrid', profile.get_int('extid')))
+        account.add_child(Node.s32('tpc', statistics.get_int('total_plays', 0)))
+        account.add_child(Node.s32('dpc', today_count))
+        account.add_child(Node.s32('crd', 1))
+        account.add_child(Node.s32('brd', 1))
+        account.add_child(Node.s32('tdc', statistics.get_int('total_days', 0)))
+        account.add_child(Node.s32('intrvld', 0))
+        account.add_child(Node.s16('ver', 0))
+        account.add_child(Node.u64('pst', 0))
+        account.add_child(Node.u64('st', Time.now() * 1000))
+        account.add_child(Node.bool('succeed', succeeded))
+        account.add_child(Node.s32('opc', 0))
+        account.add_child(Node.s32('lpc', 0))
+        account.add_child(Node.s32('cpc', 0))
+
+        # Base profile info
+        base = Node.void('base')
+        pdata.add_child(base)
+        base.add_child(Node.string('name', profile.get_str('name')))
+        base.add_child(Node.s32('mg', profile.get_int('mg')))
+        base.add_child(Node.s32('ap', profile.get_int('ap')))
+        base.add_child(Node.string('cmnt', ''))
+        base.add_child(Node.s32('uattr', profile.get_int('uattr')))
+        base.add_child(Node.s32('money', profile.get_int('money')))
+        base.add_child(Node.s32('tbs', -1))
+        base.add_child(Node.s32_array('tbgs', [-1, -1, -1, -1]))
+        base.add_child(Node.s16_array('mlog', [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]))
+        base.add_child(Node.s32('class', profile.get_int('class')))
+        base.add_child(Node.s32('class_ar', profile.get_int('class_ar')))
+
+        # Rivals
+        rival = Node.void('rival')
+        pdata.add_child(rival)
+        slotid = 0
+        for link in links:
+            if link.type != 'rival':
+                continue
+
+            rprofile = self.get_profile(link.other_userid)
+            if rprofile is None:
+                continue
+            lobbyinfo = self.data.local.lobby.get_play_session_info(self.game, self.version, link.other_userid)
+            if lobbyinfo is None:
+                lobbyinfo = ValidatedDict()
+
+            r = Node.void('r')
+            rival.add_child(r)
+            r.add_child(Node.s32('slot_id', slotid))
+            r.add_child(Node.s32('id', rprofile.get_int('extid')))
+            r.add_child(Node.string('name', rprofile.get_str('name')))
+            r.add_child(Node.s32('icon', rprofile.get_dict('config').get_int('icon_id')))
+            r.add_child(Node.s32('class', rprofile.get_int('class')))
+            r.add_child(Node.s32('class_ar', rprofile.get_int('class_ar')))
+            r.add_child(Node.bool('friend', True))
+            r.add_child(Node.bool('target', False))
+            r.add_child(Node.u32('time', lobbyinfo.get_int('time')))
+            r.add_child(Node.u8_array('ga', lobbyinfo.get_int_array('ga', 4)))
+            r.add_child(Node.u16('gp', lobbyinfo.get_int('gp')))
+            r.add_child(Node.u8_array('ipn', lobbyinfo.get_int_array('la', 4)))
+            r.add_child(Node.u8_array('pnid', lobbyinfo.get_int_array('pnid', 16)))
+            slotid = slotid + 1
+
+        # Configuration
+        configdict = profile.get_dict('config')
+        config = Node.void('config')
+        pdata.add_child(config)
+        config.add_child(Node.u8('msel_bgm', configdict.get_int('msel_bgm')))
+        config.add_child(Node.u8('narrowdown_type', configdict.get_int('narrowdown_type')))
+        config.add_child(Node.s16('icon_id', configdict.get_int('icon_id')))
+        config.add_child(Node.s16('byword_0', configdict.get_int('byword_0')))
+        config.add_child(Node.s16('byword_1', configdict.get_int('byword_1')))
+        config.add_child(Node.bool('is_auto_byword_0', configdict.get_bool('is_auto_byword_0')))
+        config.add_child(Node.bool('is_auto_byword_1', configdict.get_bool('is_auto_byword_1')))
+        config.add_child(Node.u8('mrec_type', configdict.get_int('mrec_type')))
+        config.add_child(Node.u8('tab_sel', configdict.get_int('tab_sel')))
+        config.add_child(Node.u8('card_disp', configdict.get_int('card_disp')))
+        config.add_child(Node.u8('score_tab_disp', configdict.get_int('score_tab_disp')))
+        config.add_child(Node.s16('last_music_id', configdict.get_int('last_music_id', -1)))
+        config.add_child(Node.u8('last_note_grade', configdict.get_int('last_note_grade')))
+        config.add_child(Node.u8('sort_type', configdict.get_int('sort_type')))
+        config.add_child(Node.u8('rival_panel_type', configdict.get_int('rival_panel_type')))
+        config.add_child(Node.u64('random_entry_work', configdict.get_int('random_entry_work')))
+        config.add_child(Node.u64('custom_folder_work', configdict.get_int('custom_folder_work')))
+        config.add_child(Node.u8('folder_type', configdict.get_int('folder_type')))
+        config.add_child(Node.u8('folder_lamp_type', configdict.get_int('folder_lamp_type')))
+        config.add_child(Node.bool('is_tweet', configdict.get_bool('is_tweet')))
+        config.add_child(Node.bool('is_link_twitter', configdict.get_bool('is_link_twitter')))
+
+        # Customizations
+        customdict = profile.get_dict('custom')
+        custom = Node.void('custom')
+        pdata.add_child(custom)
+        custom.add_child(Node.u8('st_shot', customdict.get_int('st_shot')))
+        custom.add_child(Node.u8('st_frame', customdict.get_int('st_frame')))
+        custom.add_child(Node.u8('st_expl', customdict.get_int('st_expl')))
+        custom.add_child(Node.u8('st_bg', customdict.get_int('st_bg')))
+        custom.add_child(Node.u8('st_shot_vol', customdict.get_int('st_shot_vol')))
+        custom.add_child(Node.u8('st_bg_bri', customdict.get_int('st_bg_bri')))
+        custom.add_child(Node.u8('st_obj_size', customdict.get_int('st_obj_size')))
+        custom.add_child(Node.u8('st_jr_gauge', customdict.get_int('st_jr_gauge')))
+        custom.add_child(Node.u8('st_clr_gauge', customdict.get_int('st_clr_gauge')))
+        custom.add_child(Node.u8('st_jdg_disp', customdict.get_int('st_jdg_disp')))
+        custom.add_child(Node.u8('st_rnd', customdict.get_int('st_rnd')))
+        custom.add_child(Node.u8('st_hazard', customdict.get_int('st_hazard')))
+        custom.add_child(Node.u8('st_clr_cond', customdict.get_int('st_clr_cond')))
+        custom.add_child(Node.u8('same_time_note_disp', customdict.get_int('same_time_note_disp')))
+        custom.add_child(Node.u8('st_gr_gauge_type', customdict.get_int('st_gr_gauge_type')))
+        custom.add_child(Node.s16('voice_message_set', customdict.get_int('voice_message_set', -1)))
+        custom.add_child(Node.u8('voice_message_volume', customdict.get_int('voice_message_volume')))
+
+        # Unlocks
+        released = Node.void('released')
+        pdata.add_child(released)
+
+        for item in achievements:
+            if item.type[:5] != 'item_':
+                continue
+            itemtype = int(item.type[5:])
+            if game_config.get_bool('force_unlock_songs') and itemtype == 0:
+                # Don't echo unlocks when we're force unlocking, we'll do it later
+                continue
+
+            info = Node.void('info')
+            released.add_child(info)
+            info.add_child(Node.u8('type', itemtype))
+            info.add_child(Node.u16('id', item.id))
+            info.add_child(Node.u16('param', item.data.get_int('param')))
+
+        if game_config.get_bool('force_unlock_songs'):
+            ids: Dict[int, int] = {}
+            songs = self.data.local.music.get_all_songs(self.game, self.version)
+            for song in songs:
+                if song.id not in ids:
+                    ids[song.id] = 0
+
+                if song.data.get_int('difficulty') > 0:
+                    ids[song.id] = ids[song.id] | (1 << song.chart)
+
+            for songid in ids:
+                if ids[songid] == 0:
+                    continue
+
+                info = Node.void('info')
+                released.add_child(info)
+                info.add_child(Node.u8('type', 0))
+                info.add_child(Node.u16('id', songid))
+                info.add_child(Node.u16('param', ids[songid]))
+
+        # Announcements
+        announce = Node.void('announce')
+        pdata.add_child(announce)
+
+        for announcement in achievements:
+            if announcement.type[:13] != 'announcement_':
+                continue
+            announcementtype = int(announcement.type[13:])
+
+            info = Node.void('info')
+            announce.add_child(info)
+            info.add_child(Node.u8('type', announcementtype))
+            info.add_child(Node.u16('id', announcement.id))
+            info.add_child(Node.u16('param', announcement.data.get_int('param')))
+            info.add_child(Node.bool('bneedannounce', announcement.data.get_bool('need')))
+
+        # Dojo ranking return
+        dojo = Node.void('dojo')
+        pdata.add_child(dojo)
+
+        for entry in achievements:
+            if entry.type != 'dojo':
+                continue
+
+            rec = Node.void('rec')
+            dojo.add_child(rec)
+            rec.add_child(Node.s32('class', entry.id))
+            rec.add_child(Node.s32('clear_type', entry.data.get_int('clear_type')))
+            rec.add_child(Node.s32('total_ar', entry.data.get_int('ar')))
+            rec.add_child(Node.s32('total_score', entry.data.get_int('score')))
+            rec.add_child(Node.s32('play_count', entry.data.get_int('plays')))
+            rec.add_child(Node.s32('last_play_time', entry.data.get_int('play_timestamp')))
+            rec.add_child(Node.s32('record_update_time', entry.data.get_int('record_timestamp')))
+            rec.add_child(Node.s32('rank', 0))
+
+        # Player Parameters
+        player_param = Node.void('player_param')
+        pdata.add_child(player_param)
+
+        for param in achievements:
+            if param.type[:13] != 'player_param_':
+                continue
+            itemtype = int(param.type[13:])
+
+            itemnode = Node.void('item')
+            player_param.add_child(itemnode)
+            itemnode.add_child(Node.s32('type', itemtype))
+            itemnode.add_child(Node.s32('bank', param.id))
+            itemnode.add_child(Node.s32_array('data', param.data.get_int_array('data', 256)))
+
+        # Shop score for players
+        self._add_shop_score(pdata)
+
+        # My List data
+        mylist = Node.void('mylist')
+        pdata.add_child(mylist)
+        listdata = Node.void('list')
+        mylist.add_child(listdata)
+        listdata.add_child(Node.s16('idx', 0))
+        listdata.add_child(Node.s16_array('mlst', profile.get_int_array('favorites', 30, [-1] * 30)))
+
+        # Minigame settings
+        minigame = Node.void('minigame')
+        pdata.add_child(minigame)
+        minigame.add_child(Node.s8('mgid', profile.get_int('mgid')))
+        minigame.add_child(Node.s32('sc', profile.get_int('mgsc')))
+
+        # Derby settings
+        derby = Node.void('derby')
+        pdata.add_child(derby)
+        derby.add_child(Node.bool('is_open', False))
+
+        return root
+
+    def unformat_profile(self, userid: UserID, request: Node, oldprofile: ValidatedDict) -> ValidatedDict:
+        game_config = self.get_game_config()
+        newprofile = copy.deepcopy(oldprofile)
+
+        # Save base player profile info
+        newprofile.replace_int('lid', ID.parse_machine_id(request.child_value('pdata/account/lid')))
+        newprofile.replace_str('name', request.child_value('pdata/base/name'))
+        newprofile.replace_int('mg', request.child_value('pdata/base/mg'))
+        newprofile.replace_int('ap', request.child_value('pdata/base/ap'))
+        newprofile.replace_int('uattr', request.child_value('pdata/base/uattr'))
+        newprofile.replace_int('money', request.child_value('pdata/base/money'))
+        newprofile.replace_int('class', request.child_value('pdata/base/class'))
+        newprofile.replace_int('class_ar', request.child_value('pdata/base/class_ar'))
+        newprofile.replace_int('mgid', request.child_value('pdata/minigame/mgid'))
+        newprofile.replace_int('mgsc', request.child_value('pdata/minigame/sc'))
+        newprofile.replace_int_array('favorites', 30, request.child_value('pdata/mylist/list/mlst'))
+
+        # Save player config
+        configdict = newprofile.get_dict('config')
+        config = request.child('pdata/config')
+        if config:
+            configdict.replace_int('msel_bgm', config.child_value('msel_bgm'))
+            configdict.replace_int('narrowdown_type', config.child_value('narrowdown_type'))
+            configdict.replace_int('icon_id', config.child_value('icon_id'))
+            configdict.replace_int('byword_0', config.child_value('byword_0'))
+            configdict.replace_int('byword_1', config.child_value('byword_1'))
+            configdict.replace_bool('is_auto_byword_0', config.child_value('is_auto_byword_0'))
+            configdict.replace_bool('is_auto_byword_1', config.child_value('is_auto_byword_1'))
+            configdict.replace_int('mrec_type', config.child_value('mrec_type'))
+            configdict.replace_int('tab_sel', config.child_value('tab_sel'))
+            configdict.replace_int('card_disp', config.child_value('card_disp'))
+            configdict.replace_int('score_tab_disp', config.child_value('score_tab_disp'))
+            configdict.replace_int('last_music_id', config.child_value('last_music_id'))
+            configdict.replace_int('last_note_grade', config.child_value('last_note_grade'))
+            configdict.replace_int('sort_type', config.child_value('sort_type'))
+            configdict.replace_int('rival_panel_type', config.child_value('rival_panel_type'))
+            configdict.replace_int('random_entry_work', config.child_value('random_entry_work'))
+            configdict.replace_int('custom_folder_work', config.child_value('custom_folder_work'))
+            configdict.replace_int('folder_type', config.child_value('folder_type'))
+            configdict.replace_int('folder_lamp_type', config.child_value('folder_lamp_type'))
+            configdict.replace_bool('is_tweet', config.child_value('is_tweet'))
+            configdict.replace_bool('is_link_twitter', config.child_value('is_link_twitter'))
+        newprofile.replace_dict('config', configdict)
+
+        # Save player custom settings
+        customdict = newprofile.get_dict('custom')
+        custom = request.child('pdata/custom')
+        if custom:
+            customdict.replace_int('st_shot', custom.child_value('st_shot'))
+            customdict.replace_int('st_frame', custom.child_value('st_frame'))
+            customdict.replace_int('st_expl', custom.child_value('st_expl'))
+            customdict.replace_int('st_bg', custom.child_value('st_bg'))
+            customdict.replace_int('st_shot_vol', custom.child_value('st_shot_vol'))
+            customdict.replace_int('st_bg_bri', custom.child_value('st_bg_bri'))
+            customdict.replace_int('st_obj_size', custom.child_value('st_obj_size'))
+            customdict.replace_int('st_jr_gauge', custom.child_value('st_jr_gauge'))
+            customdict.replace_int('st_clr_gauge', custom.child_value('st_clr_gauge'))
+            customdict.replace_int('st_rnd', custom.child_value('st_rnd'))
+            customdict.replace_int('st_hazard', custom.child_value('st_hazard'))
+            customdict.replace_int('st_clr_cond', custom.child_value('st_clr_cond'))
+            customdict.replace_int('same_time_note_disp', custom.child_value('same_time_note_disp'))
+            customdict.replace_int('st_gr_gauge_type', custom.child_value('st_gr_gauge_type'))
+            customdict.replace_int('voice_message_set', custom.child_value('voice_message_set'))
+            customdict.replace_int('voice_message_volume', custom.child_value('voice_message_volume'))
+        newprofile.replace_dict('custom', customdict)
+
+        # Save player parameter info
+        params = request.child('pdata/player_param')
+        if params:
+            for child in params.children:
+                if child.name != 'item':
+                    continue
+
+                item_type = child.child_value('type')
+                bank = child.child_value('bank')
+                data = child.child_value('data')
+                while len(data) < 256:
+                    data.append(0)
+                self.data.local.user.put_achievement(
+                    self.game,
+                    self.version,
+                    userid,
+                    bank,
+                    'player_param_{}'.format(item_type),
+                    {
+                        'data': data,
+                    },
+                )
+
+        # Save player episode info
+        episode = request.child('pdata/episode')
+        if episode:
+            for child in episode.children:
+                if child.name != 'info':
+                    continue
+
+                # I assume this is copypasta, but I want to be sure
+                extid = child.child_value('user_id')
+                if extid != newprofile.get_int('extid'):
+                    raise Exception('Unexpected user ID, got {} expecting {}'.format(extid, newprofile.get_int('extid')))
+
+                episode_type = child.child_value('type')
+                episode_value0 = child.child_value('value0')
+                episode_value1 = child.child_value('value1')
+                episode_text = child.child_value('text')
+                episode_time = child.child_value('time')
+                self.data.local.user.put_achievement(
+                    self.game,
+                    self.version,
+                    userid,
+                    episode_type,
+                    'episode',
+                    {
+                        'value0': episode_value0,
+                        'value1': episode_value1,
+                        'text': episode_text,
+                        'time': episode_time,
+                    },
+                )
+
+        # Save released info
+        released = request.child('pdata/released')
+        if released:
+            for child in released.children:
+                if child.name != 'info':
+                    continue
+
+                item_id = child.child_value('id')
+                item_type = child.child_value('type')
+                param = child.child_value('param')
+                if game_config.get_bool('force_unlock_songs') and item_type == 0:
+                    # Don't save unlocks when we're force unlocking
+                    continue
+
+                self.data.local.user.put_achievement(
+                    self.game,
+                    self.version,
+                    userid,
+                    item_id,
+                    'item_{}'.format(item_type),
+                    {
+                        'param': param,
+                    },
+                )
+
+        # Save announce info
+        announce = request.child('pdata/announce')
+        if announce:
+            for child in announce.children:
+                if child.name != 'info':
+                    continue
+
+                announce_id = child.child_value('id')
+                announce_type = child.child_value('type')
+                param = child.child_value('param')
+                need = child.child_value('bneedannounce')
+                self.data.local.user.put_achievement(
+                    self.game,
+                    self.version,
+                    userid,
+                    announce_id,
+                    'announcement_{}'.format(announce_type),
+                    {
+                        'param': param,
+                        'need': need,
+                    },
+                )
+
+        # Save player dojo
+        dojo = request.child('pdata/dojo')
+        if dojo:
+            dojoid = dojo.child_value('class')
+            clear_type = dojo.child_value('clear_type')
+            ar = dojo.child_value('t_ar')
+            score = dojo.child_value('t_score')
+
+            # Figure out timestamp stuff
+            data = self.data.local.user.get_achievement(
+                self.game,
+                self.version,
+                userid,
+                dojoid,
+                'dojo',
+            ) or ValidatedDict()
+
+            if ar >= data.get_int('ar'):
+                # We set a new achievement rate, keep the new values
+                record_time = Time.now()
+            else:
+                # We didn't, keep the old values for achievement rate, but
+                # override score and clear_type only if they were better.
+                record_time = data.get_int('record_timestamp')
+                ar = data.get_int('ar')
+                score = max(score, data.get_int('score'))
+                clear_type = max(clear_type, data.get_int('clear_type'))
+
+            play_time = Time.now()
+            plays = data.get_int('plays') + 1
+
+            self.data.local.user.put_achievement(
+                self.game,
+                self.version,
+                userid,
+                dojoid,
+                'dojo',
+                {
+                    'clear_type': clear_type,
+                    'ar': ar,
+                    'score': score,
+                    'plays': plays,
+                    'play_timestamp': play_time,
+                    'record_timestamp': record_time,
+                },
+            )
+
+        # Grab any new rivals added during this play session
+        rivalnode = request.child('pdata/rival')
+        if rivalnode:
+            for child in rivalnode.children:
+                if child.name != 'r':
+                    continue
+
+                extid = child.child_value('id')
+                other_userid = self.data.remote.user.from_extid(self.game, self.version, extid)
+                if other_userid is None:
+                    continue
+
+                self.data.local.user.put_link(
+                    self.game,
+                    self.version,
+                    userid,
+                    'rival',
+                    other_userid,
+                    {},
+                )
+
+        # Grab any new records set during this play session
+        songplays = request.child('pdata/stglog')
+        if songplays:
+            for child in songplays.children:
+                if child.name != 'log':
+                    continue
+
+                songid = child.child_value('mid')
+                chart = child.child_value('ng')
+                clear_type = child.child_value('ct')
+                if songid == 0 and chart == 0 and clear_type == -1:
+                    # Dummy song save during profile create
+                    continue
+
+                points = child.child_value('sc')
+                achievement_rate = child.child_value('ar')
+                param = child.child_value('param')
+                miss_count = child.child_value('jt_ms')
+                k_flag = child.child_value('k_flag')
+
+                # Param is some random bits along with the combo type
+                combo_type = param & 0x3
+                param = param ^ combo_type
+
+                clear_type = self._game_to_db_clear_type(clear_type)
+                combo_type = self._game_to_db_combo_type(combo_type, miss_count)
+                self.update_score(
+                    userid,
+                    songid,
+                    chart,
+                    points,
+                    achievement_rate,
+                    clear_type,
+                    combo_type,
+                    miss_count,
+                    param=param,
+                    kflag=k_flag,
+                )
+
+        # Keep track of play statistics
+        self.update_play_statistics(userid)
+
+        return newprofile

bemani/backend/reflec/volzzabase.py

@@ -0,0 +1,504 @@
+from typing import Dict, List, Tuple
+
+from bemani.backend.reflec.base import ReflecBeatBase
+
+from bemani.common import ID, Time, ValidatedDict
+from bemani.data import Attempt, UserID
+from bemani.protocol import Node
+
+
+class ReflecBeatVolzzaBase(ReflecBeatBase):
+
+    # Clear types according to the game
+    GAME_CLEAR_TYPE_NO_PLAY = 0
+    GAME_CLEAR_TYPE_EARLY_FAILED = 1
+    GAME_CLEAR_TYPE_FAILED = 2
+    GAME_CLEAR_TYPE_CLEARED = 9
+    GAME_CLEAR_TYPE_HARD_CLEARED = 10
+    GAME_CLEAR_TYPE_S_HARD_CLEARED = 11
+
+    # Combo types according to the game (actually a bitmask, where bit 0 is
+    # full combo status, and bit 2 is just reflec status). But we don't support
+    # saving just reflec without full combo, so we downgrade it.
+    GAME_COMBO_TYPE_NONE = 0
+    GAME_COMBO_TYPE_ALL_JUST = 2
+    GAME_COMBO_TYPE_FULL_COMBO = 1
+    GAME_COMBO_TYPE_FULL_COMBO_ALL_JUST = 3
+
+    def _db_to_game_clear_type(self, db_status: int) -> int:
+        return {
+            self.CLEAR_TYPE_NO_PLAY: self.GAME_CLEAR_TYPE_NO_PLAY,
+            self.CLEAR_TYPE_FAILED: self.GAME_CLEAR_TYPE_FAILED,
+            self.CLEAR_TYPE_CLEARED: self.GAME_CLEAR_TYPE_CLEARED,
+            self.CLEAR_TYPE_HARD_CLEARED: self.GAME_CLEAR_TYPE_HARD_CLEARED,
+            self.CLEAR_TYPE_S_HARD_CLEARED: self.GAME_CLEAR_TYPE_S_HARD_CLEARED,
+        }[db_status]
+
+    def _game_to_db_clear_type(self, status: int) -> int:
+        return {
+            self.GAME_CLEAR_TYPE_NO_PLAY: self.CLEAR_TYPE_NO_PLAY,
+            self.GAME_CLEAR_TYPE_EARLY_FAILED: self.CLEAR_TYPE_FAILED,
+            self.GAME_CLEAR_TYPE_FAILED: self.CLEAR_TYPE_FAILED,
+            self.GAME_CLEAR_TYPE_CLEARED: self.CLEAR_TYPE_CLEARED,
+            self.GAME_CLEAR_TYPE_HARD_CLEARED: self.CLEAR_TYPE_HARD_CLEARED,
+            self.GAME_CLEAR_TYPE_S_HARD_CLEARED: self.CLEAR_TYPE_S_HARD_CLEARED,
+        }[status]
+
+    def _db_to_game_combo_type(self, db_combo: int) -> int:
+        return {
+            self.COMBO_TYPE_NONE: self.GAME_COMBO_TYPE_NONE,
+            self.COMBO_TYPE_ALMOST_COMBO: self.GAME_COMBO_TYPE_NONE,
+            self.COMBO_TYPE_FULL_COMBO: self.GAME_COMBO_TYPE_FULL_COMBO,
+            self.COMBO_TYPE_FULL_COMBO_ALL_JUST: self.GAME_COMBO_TYPE_FULL_COMBO_ALL_JUST,
+        }[db_combo]
+
+    def _game_to_db_combo_type(self, game_combo: int, miss_count: int) -> int:
+        if game_combo in [
+            self.GAME_COMBO_TYPE_NONE,
+            self.GAME_COMBO_TYPE_ALL_JUST,
+        ]:
+            if miss_count >= 0 and miss_count <= 2:
+                return self.COMBO_TYPE_ALMOST_COMBO
+            else:
+                return self.COMBO_TYPE_NONE
+        if game_combo == self.GAME_COMBO_TYPE_FULL_COMBO:
+            return self.COMBO_TYPE_FULL_COMBO
+        if game_combo == self.GAME_COMBO_TYPE_FULL_COMBO_ALL_JUST:
+            return self.COMBO_TYPE_FULL_COMBO_ALL_JUST
+        raise Exception('Invalid game_combo value {}'.format(game_combo))
+
+    def _add_event_info(self, root: Node) -> None:
+        # Overridden in subclasses
+        pass
+
+    def _add_shop_score(self, root: Node) -> None:
+        shop_score = Node.void('shop_score')
+        root.add_child(shop_score)
+        today = Node.void('today')
+        shop_score.add_child(today)
+        yesterday = Node.void('yesterday')
+        shop_score.add_child(yesterday)
+
+        all_profiles = self.data.local.user.get_all_profiles(self.game, self.version)
+        all_attempts = self.data.local.music.get_all_attempts(self.game, self.version, timelimit=(Time.beginning_of_today() - Time.SECONDS_IN_DAY))
+        machine = self.data.local.machine.get_machine(self.config['machine']['pcbid'])
+        if machine.arcade is not None:
+            lids = [
+                machine.id for machine in self.data.local.machine.get_all_machines(machine.arcade)
+            ]
+        else:
+            lids = [machine.id]
+
+        relevant_profiles = [
+            profile for profile in all_profiles
+            if profile[1].get_int('lid', -1) in lids
+        ]
+
+        for (rootnode, timeoffset) in [
+            (today, 0),
+            (yesterday, Time.SECONDS_IN_DAY),
+        ]:
+            # Grab all attempts made in the relevant day
+            relevant_attempts = [
+                attempt for attempt in all_attempts
+                if (
+                    attempt[1].timestamp >= (Time.beginning_of_today() - timeoffset) and
+                    attempt[1].timestamp <= (Time.end_of_today() - timeoffset)
+                )
+            ]
+
+            # Calculate scores based on attempt
+            scores_by_user: Dict[UserID, Dict[int, Dict[int, Attempt]]] = {}
+            for (userid, attempt) in relevant_attempts:
+                if userid not in scores_by_user:
+                    scores_by_user[userid] = {}
+                if attempt.id not in scores_by_user[userid]:
+                    scores_by_user[userid][attempt.id] = {}
+                if attempt.chart not in scores_by_user[userid][attempt.id]:
+                    # No high score for this yet, just use this attempt
+                    scores_by_user[userid][attempt.id][attempt.chart] = attempt
+                else:
+                    # If this attempt is better than the stored one, replace it
+                    if scores_by_user[userid][attempt.id][attempt.chart].points < attempt.points:
+                        scores_by_user[userid][attempt.id][attempt.chart] = attempt
+
+            # Calculate points earned by user in the day
+            points_by_user: Dict[UserID, int] = {}
+            for userid in scores_by_user:
+                points_by_user[userid] = 0
+                for mid in scores_by_user[userid]:
+                    for chart in scores_by_user[userid][mid]:
+                        points_by_user[userid] = points_by_user[userid] + scores_by_user[userid][mid][chart].points
+
+            # Output that day's earned points
+            for (userid, profile) in relevant_profiles:
+                data = Node.void('data')
+                rootnode.add_child(data)
+                data.add_child(Node.s16('day_id', int((Time.now() - timeoffset) / Time.SECONDS_IN_DAY)))
+                data.add_child(Node.s32('user_id', profile.get_int('extid')))
+                data.add_child(Node.s16('icon_id', profile.get_dict('config').get_int('icon_id')))
+                data.add_child(Node.s16('point', min(points_by_user.get(userid, 0), 32767)))
+                data.add_child(Node.s32('update_time', Time.now()))
+                data.add_child(Node.string('name', profile.get_str('name')))
+
+            rootnode.add_child(Node.s32('time', Time.beginning_of_today() - timeoffset))
+
+    def handle_info_rb5_info_read_request(self, request: Node) -> Node:
+        root = Node.void('info')
+        self._add_event_info(root)
+
+        return root
+
+    def handle_info_rb5_info_read_hit_chart_request(self, request: Node) -> Node:
+        version = request.child_value('ver')
+
+        root = Node.void('info')
+        root.add_child(Node.s32('ver', version))
+        ranking = Node.void('ranking')
+        root.add_child(ranking)
+
+        def add_hitchart(name: str, start: int, end: int, hitchart: List[Tuple[int, int]]) -> None:
+            base = Node.void(name)
+            ranking.add_child(base)
+            base.add_child(Node.s32('bt', start))
+            base.add_child(Node.s32('et', end))
+            new = Node.void('new')
+            base.add_child(new)
+
+            for (mid, plays) in hitchart:
+                d = Node.void('d')
+                new.add_child(d)
+                d.add_child(Node.s16('mid', mid))
+                d.add_child(Node.s32('cnt', plays))
+
+        # Weekly hit chart
+        add_hitchart(
+            'weekly',
+            Time.now() - Time.SECONDS_IN_WEEK,
+            Time.now(),
+            self.data.local.music.get_hit_chart(self.game, self.version, 1024, 7),
+        )
+
+        # Monthly hit chart
+        add_hitchart(
+            'monthly',
+            Time.now() - Time.SECONDS_IN_DAY * 30,
+            Time.now(),
+            self.data.local.music.get_hit_chart(self.game, self.version, 1024, 30),
+        )
+
+        # All time hit chart
+        add_hitchart(
+            'total',
+            Time.now() - Time.SECONDS_IN_DAY * 365,
+            Time.now(),
+            self.data.local.music.get_hit_chart(self.game, self.version, 1024, 365),
+        )
+
+        return root
+
+    def handle_info_rb5_info_read_shop_ranking_request(self, request: Node) -> Node:
+        start_music_id = request.child_value('min')
+        end_music_id = request.child_value('max')
+
+        root = Node.void('info')
+        shop_score = Node.void('shop_score')
+        root.add_child(shop_score)
+        shop_score.add_child(Node.s32('time', Time.now()))
+
+        profiles: Dict[UserID, ValidatedDict] = {}
+        for songid in range(start_music_id, end_music_id + 1):
+            allscores = self.data.local.music.get_all_scores(
+                self.game,
+                self.version,
+                songid=songid,
+            )
+
+            for ng in [
+                self.CHART_TYPE_BASIC,
+                self.CHART_TYPE_MEDIUM,
+                self.CHART_TYPE_HARD,
+                self.CHART_TYPE_SPECIAL,
+            ]:
+                scores = sorted(
+                    [score for score in allscores if score[1].chart == ng],
+                    key=lambda score: score[1].points,
+                    reverse=True,
+                )
+
+                for i in range(len(scores)):
+                    userid, score = scores[i]
+                    if userid not in profiles:
+                        profiles[userid] = self.get_any_profile(userid)
+                    profile = profiles[userid]
+
+                    data = Node.void('data')
+                    shop_score.add_child(data)
+                    data.add_child(Node.s32('rank', i + 1))
+                    data.add_child(Node.s16('music_id', songid))
+                    data.add_child(Node.s8('note_grade', score.chart))
+                    data.add_child(Node.s8('clear_type', self._db_to_game_clear_type(score.data.get_int('clear_type'))))
+                    data.add_child(Node.s32('user_id', profile.get_int('extid')))
+                    data.add_child(Node.s16('icon_id', profile.get_dict('config').get_int('icon_id')))
+                    data.add_child(Node.s32('score', score.points))
+                    data.add_child(Node.s32('time', score.timestamp))
+                    data.add_child(Node.string('name', profile.get_str('name')))
+
+        return root
+
+    def handle_lobby_rb5_lobby_entry_request(self, request: Node) -> Node:
+        root = Node.void('lobby')
+        root.add_child(Node.s32('interval', 120))
+        root.add_child(Node.s32('interval_p', 120))
+
+        # Create a lobby entry for this user
+        extid = request.child_value('e/uid')
+        userid = self.data.remote.user.from_extid(self.game, self.version, extid)
+        if userid is not None:
+            profile = self.get_profile(userid)
+            info = self.data.local.lobby.get_play_session_info(self.game, self.version, userid)
+            if profile is None or info is None:
+                return root
+
+            self.data.local.lobby.put_lobby(
+                self.game,
+                self.version,
+                userid,
+                {
+                    'mid': request.child_value('e/mid'),
+                    'ng': request.child_value('e/ng'),
+                    'mopt': request.child_value('e/mopt'),
+                    'lid': request.child_value('e/lid'),
+                    'sn': request.child_value('e/sn'),
+                    'pref': request.child_value('e/pref'),
+                    'stg': request.child_value('e/stg'),
+                    'pside': request.child_value('e/pside'),
+                    'eatime': request.child_value('e/eatime'),
+                    'ga': request.child_value('e/ga'),
+                    'gp': request.child_value('e/gp'),
+                    'la': request.child_value('e/la'),
+                    'ver': request.child_value('e/ver'),
+                }
+            )
+            lobby = self.data.local.lobby.get_lobby(
+                self.game,
+                self.version,
+                userid,
+            )
+            root.add_child(Node.s32('eid', lobby.get_int('id')))
+            e = Node.void('e')
+            root.add_child(e)
+            e.add_child(Node.s32('eid', lobby.get_int('id')))
+            e.add_child(Node.u16('mid', lobby.get_int('mid')))
+            e.add_child(Node.u8('ng', lobby.get_int('ng')))
+            e.add_child(Node.s32('uid', profile.get_int('extid')))
+            e.add_child(Node.s32('uattr', profile.get_int('uattr')))
+            e.add_child(Node.string('pn', profile.get_str('name')))
+            e.add_child(Node.s32('plyid', info.get_int('id')))
+            e.add_child(Node.s16('mg', profile.get_int('mg')))
+            e.add_child(Node.s32('mopt', lobby.get_int('mopt')))
+            e.add_child(Node.string('lid', lobby.get_str('lid')))
+            e.add_child(Node.string('sn', lobby.get_str('sn')))
+            e.add_child(Node.u8('pref', lobby.get_int('pref')))
+            e.add_child(Node.s8('stg', lobby.get_int('stg')))
+            e.add_child(Node.s8('pside', lobby.get_int('pside')))
+            e.add_child(Node.s16('eatime', lobby.get_int('eatime')))
+            e.add_child(Node.u8_array('ga', lobby.get_int_array('ga', 4)))
+            e.add_child(Node.u16('gp', lobby.get_int('gp')))
+            e.add_child(Node.u8_array('la', lobby.get_int_array('la', 4)))
+            e.add_child(Node.u8('ver', lobby.get_int('ver')))
+
+        return root
+
+    def handle_lobby_rb5_lobby_read_request(self, request: Node) -> Node:
+        root = Node.void('lobby')
+        root.add_child(Node.s32('interval', 120))
+        root.add_child(Node.s32('interval_p', 120))
+
+        # Look up all lobbies matching the criteria specified
+        ver = request.child_value('var')
+        mg = request.child_value('m_grade')  # noqa: F841
+        extid = request.child_value('uid')
+        limit = request.child_value('max')
+        userid = self.data.remote.user.from_extid(self.game, self.version, extid)
+        if userid is not None:
+            lobbies = self.data.local.lobby.get_all_lobbies(self.game, self.version)
+            for (user, lobby) in lobbies:
+                if limit <= 0:
+                    break
+
+                if user == userid:
+                    # If we have our own lobby, don't return it
+                    continue
+                if ver != lobby.get_int('ver'):
+                    # Don't return lobby data for different versions
+                    continue
+
+                profile = self.get_profile(user)
+                info = self.data.local.lobby.get_play_session_info(self.game, self.version, userid)
+                if profile is None or info is None:
+                    # No profile info, don't return this lobby
+                    return root
+
+                e = Node.void('e')
+                root.add_child(e)
+                e.add_child(Node.s32('eid', lobby.get_int('id')))
+                e.add_child(Node.u16('mid', lobby.get_int('mid')))
+                e.add_child(Node.u8('ng', lobby.get_int('ng')))
+                e.add_child(Node.s32('uid', profile.get_int('extid')))
+                e.add_child(Node.s32('uattr', profile.get_int('uattr')))
+                e.add_child(Node.string('pn', profile.get_str('name')))
+                e.add_child(Node.s32('plyid', info.get_int('id')))
+                e.add_child(Node.s16('mg', profile.get_int('mg')))
+                e.add_child(Node.s32('mopt', lobby.get_int('mopt')))
+                e.add_child(Node.string('lid', lobby.get_str('lid')))
+                e.add_child(Node.string('sn', lobby.get_str('sn')))
+                e.add_child(Node.u8('pref', lobby.get_int('pref')))
+                e.add_child(Node.s8('stg', lobby.get_int('stg')))
+                e.add_child(Node.s8('pside', lobby.get_int('pside')))
+                e.add_child(Node.s16('eatime', lobby.get_int('eatime')))
+                e.add_child(Node.u8_array('ga', lobby.get_int_array('ga', 4)))
+                e.add_child(Node.u16('gp', lobby.get_int('gp')))
+                e.add_child(Node.u8_array('la', lobby.get_int_array('la', 4)))
+                e.add_child(Node.u8('ver', lobby.get_int('ver')))
+
+                limit = limit - 1
+
+        return root
+
+    def handle_lobby_rb5_lobby_delete_entry_request(self, request: Node) -> Node:
+        eid = request.child_value('eid')
+        self.data.local.lobby.destroy_lobby(eid)
+        return Node.void('lobby')
+
+    def handle_pcb_rb5_pcb_boot_request(self, request: Node) -> Node:
+        shop_id = ID.parse_machine_id(request.child_value('lid'))
+        machine = self.get_machine_by_id(shop_id)
+        if machine is not None:
+            machine_name = machine.name
+            close = machine.data.get_bool('close')
+            hour = machine.data.get_int('hour')
+            minute = machine.data.get_int('minute')
+        else:
+            machine_name = ''
+            close = False
+            hour = 0
+            minute = 0
+
+        root = Node.void('pcb')
+        sinfo = Node.void('sinfo')
+        root.add_child(sinfo)
+        sinfo.add_child(Node.string('nm', machine_name))
+        sinfo.add_child(Node.bool('cl_enbl', close))
+        sinfo.add_child(Node.u8('cl_h', hour))
+        sinfo.add_child(Node.u8('cl_m', minute))
+        sinfo.add_child(Node.bool('shop_flag', True))
+        return root
+
+    def handle_pcb_rb5_pcb_error_request(self, request: Node) -> Node:
+        return Node.void('pcb')
+
+    def handle_pcb_rb5_pcb_update_request(self, request: Node) -> Node:
+        return Node.void('pcb')
+
+    def handle_shop_rb5_shop_write_setting_request(self, request: Node) -> Node:
+        return Node.void('shop')
+
+    def handle_shop_rb5_shop_write_info_request(self, request: Node) -> Node:
+        self.update_machine_name(request.child_value('sinfo/nm'))
+        self.update_machine_data({
+            'close': request.child_value('sinfo/cl_enbl'),
+            'hour': request.child_value('sinfo/cl_h'),
+            'minute': request.child_value('sinfo/cl_m'),
+            'pref': request.child_value('sinfo/prf'),
+        })
+        return Node.void('shop')
+
+    def handle_player_rb5_player_start_request(self, request: Node) -> Node:
+        root = Node.void('player')
+
+        # Create a new play session based on info from the request
+        refid = request.child_value('rid')
+        userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        if userid is not None:
+            self.data.local.lobby.put_play_session_info(
+                self.game,
+                self.version,
+                userid,
+                {
+                    'ga': request.child_value('ga'),
+                    'gp': request.child_value('gp'),
+                    'la': request.child_value('la'),
+                    'pnid': request.child_value('pnid'),
+                },
+            )
+            info = self.data.local.lobby.get_play_session_info(
+                self.game,
+                self.version,
+                userid,
+            )
+            if info is not None:
+                play_id = info.get_int('id')
+            else:
+                play_id = 0
+        else:
+            play_id = 0
+
+        # Session stuff, and resend global defaults
+        root.add_child(Node.s32('plyid', play_id))
+        root.add_child(Node.u64('start_time', Time.now() * 1000))
+        self._add_event_info(root)
+
+        return root
+
+    def handle_player_rb5_player_end_request(self, request: Node) -> Node:
+        # Destroy play session based on info from the request
+        refid = request.child_value('rid')
+        userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        if userid is not None:
+            # Kill any lingering lobbies by this user
+            lobby = self.data.local.lobby.get_lobby(
+                self.game,
+                self.version,
+                userid,
+            )
+            if lobby is not None:
+                self.data.local.lobby.destroy_lobby(lobby.get_int('id'))
+            self.data.local.lobby.destroy_play_session_info(self.game, self.version, userid)
+
+        return Node.void('player')
+
+    def handle_player_rb5_player_delete_request(self, request: Node) -> Node:
+        return Node.void('player')
+
+    def handle_player_rb5_player_succeed_request(self, request: Node) -> Node:
+        refid = request.child_value('rid')
+        userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        if userid is not None:
+            previous_version = self.previous_version()
+            profile = previous_version.get_profile(userid)
+        else:
+            profile = None
+
+        root = Node.void('player')
+
+        if profile is None:
+            # Return empty succeed to say this is new
+            root.add_child(Node.string('name', ''))
+            root.add_child(Node.s32('grd', -1))
+            root.add_child(Node.s32('ap', -1))
+            root.add_child(Node.s32('uattr', 0))
+        else:
+            # Return previous profile formatted to say this is data succession
+            root.add_child(Node.string('name', profile.get_str('name')))
+            root.add_child(Node.s32('grd', profile.get_int('mg')))  # This is a guess
+            root.add_child(Node.s32('ap', profile.get_int('ap')))
+            root.add_child(Node.s32('uattr', profile.get_int('uattr')))
+        return root
+
+    def handle_player_rb5_player_read_request(self, request: Node) -> Node:
+        refid = request.child_value('rid')
+        profile = self.get_profile_by_refid(refid)
+        if profile:
+            return profile
+        return Node.void('player')

bemani/backend/sdvx/__init__.py

@@ -0,0 +1,2 @@
+from bemani.backend.sdvx.factory import SoundVoltexFactory
+from bemani.backend.sdvx.base import SoundVoltexBase

bemani/backend/sdvx/base.py

@@ -0,0 +1,291 @@
+# vim: set fileencoding=utf-8
+from typing import Dict, Optional
+
+from bemani.backend.base import Base
+from bemani.backend.core import CoreHandler, CardManagerHandler, PASELIHandler
+from bemani.common import ValidatedDict, GameConstants, DBConstants, Parallel
+from bemani.data import UserID
+from bemani.protocol import Node
+
+
+class SoundVoltexBase(CoreHandler, CardManagerHandler, PASELIHandler, Base):
+    """
+    Base game class for all Sound Voltex version that we support.
+    """
+
+    game = GameConstants.SDVX
+
+    CLEAR_TYPE_NO_PLAY = DBConstants.SDVX_CLEAR_TYPE_NO_PLAY
+    CLEAR_TYPE_FAILED = DBConstants.SDVX_CLEAR_TYPE_FAILED
+    CLEAR_TYPE_CLEAR = DBConstants.SDVX_CLEAR_TYPE_CLEAR
+    CLEAR_TYPE_HARD_CLEAR = DBConstants.SDVX_CLEAR_TYPE_HARD_CLEAR
+    CLEAR_TYPE_ULTIMATE_CHAIN = DBConstants.SDVX_CLEAR_TYPE_ULTIMATE_CHAIN
+    CLEAR_TYPE_PERFECT_ULTIMATE_CHAIN = DBConstants.SDVX_CLEAR_TYPE_PERFECT_ULTIMATE_CHAIN
+
+    GRADE_NO_PLAY = DBConstants.SDVX_GRADE_NO_PLAY
+    GRADE_D = DBConstants.SDVX_GRADE_D
+    GRADE_C = DBConstants.SDVX_GRADE_C
+    GRADE_B = DBConstants.SDVX_GRADE_B
+    GRADE_A = DBConstants.SDVX_GRADE_A
+    GRADE_A_PLUS = DBConstants.SDVX_GRADE_A_PLUS
+    GRADE_AA = DBConstants.SDVX_GRADE_AA
+    GRADE_AA_PLUS = DBConstants.SDVX_GRADE_AA_PLUS
+    GRADE_AAA = DBConstants.SDVX_GRADE_AAA
+    GRADE_AAA_PLUS = DBConstants.SDVX_GRADE_AAA_PLUS
+    GRADE_S = DBConstants.SDVX_GRADE_S
+
+    CHART_TYPE_NOVICE = 0
+    CHART_TYPE_ADVANCED = 1
+    CHART_TYPE_EXHAUST = 2
+    CHART_TYPE_INFINITE = 3
+    CHART_TYPE_MAXIMUM = 4
+
+    def previous_version(self) -> Optional['SoundVoltexBase']:
+        """
+        Returns the previous version of the game, based on this game. Should
+        be overridden.
+        """
+        return None
+
+    def get_profile_by_refid(self, refid: Optional[str]) -> Optional[Node]:
+        """
+        Given a RefID, return a formatted profile node. Basically every game
+        needs a profile lookup, even if it handles where that happens in
+        a different request. This is provided for code deduplication.
+        """
+        if refid is None:
+            return None
+
+        # First try to load the actual profile
+        userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        profile = self.get_profile(userid)
+        if profile is None:
+            return None
+
+        # Now, return it
+        return self.format_profile(userid, profile)
+
+    def new_profile_by_refid(self, refid: Optional[str], name: Optional[str], locid: Optional[int]) -> Node:
+        """
+        Given a RefID and an optional name, create a profile and then return
+        a formatted profile node. Similar rationale to get_profile_by_refid.
+        """
+        if refid is None:
+            return None
+
+        if name is None:
+            name = 'NONAME'
+
+        # First, create and save the default profile
+        userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        defaultprofile = ValidatedDict({
+            'name': name,
+            'loc': locid,
+        })
+        self.put_profile(userid, defaultprofile)
+
+        # Now, reload and format the profile, looking up the has old version flag
+        profile = self.get_profile(userid)
+        return self.format_profile(userid, profile)
+
+    def format_profile(self, userid: UserID, profile: ValidatedDict) -> Node:
+        """
+        Base handler for a profile. Given a userid and a profile dictionary,
+        return a Node representing a profile. Should be overridden.
+        """
+        return Node.void('game')
+
+    def unformat_profile(self, userid: UserID, request: Node, oldprofile: ValidatedDict) -> ValidatedDict:
+        """
+        Base handler for profile parsing. Given a request and an old profile,
+        return a new profile that's been updated with the contents of the request.
+        Should be overridden.
+        """
+        return oldprofile
+
+    def get_clear_rates(self) -> Dict[int, Dict[int, Dict[str, int]]]:
+        """
+        Returns a dictionary similar to the following:
+
+        {
+            musicid: {
+                chart: {
+                    total: total plays,
+                    clears: total clears,
+                    average: average score,
+                },
+            },
+        }
+        """
+        all_attempts, remote_attempts = Parallel.execute([
+            lambda: self.data.local.music.get_all_attempts(
+                game=self.game,
+                version=self.version,
+            ),
+            lambda: self.data.remote.music.get_clear_rates(
+                game=self.game,
+                version=self.version,
+            )
+        ])
+        attempts: Dict[int, Dict[int, Dict[str, int]]] = {}
+        for (_, attempt) in all_attempts:
+            # Terrible temporary structure is terrible.
+            if attempt.id not in attempts:
+                attempts[attempt.id] = {}
+            if attempt.chart not in attempts[attempt.id]:
+                attempts[attempt.id][attempt.chart] = {
+                    'total': 0,
+                    'clears': 0,
+                    'average': 0,
+                }
+
+            # We saw an attempt, keep the total attempts in sync.
+            attempts[attempt.id][attempt.chart]['average'] = int(
+                (
+                    (attempts[attempt.id][attempt.chart]['average'] * attempts[attempt.id][attempt.chart]['total']) +
+                    attempt.points
+                ) / (attempts[attempt.id][attempt.chart]['total'] + 1)
+            )
+            attempts[attempt.id][attempt.chart]['total'] += 1
+
+            if attempt.data.get_int('clear_type', self.CLEAR_TYPE_NO_PLAY) in [self.CLEAR_TYPE_NO_PLAY, self.CLEAR_TYPE_FAILED]:
+                # This attempt was a failure, so don't count it against clears of full combos
+                continue
+
+            # It was at least a clear
+            attempts[attempt.id][attempt.chart]['clears'] += 1
+
+        # Merge in remote attempts
+        for songid in remote_attempts:
+            if songid not in attempts:
+                attempts[songid] = {}
+
+            for songchart in remote_attempts[songid]:
+                if songchart not in attempts[songid]:
+                    attempts[songid][songchart] = {
+                        'total': 0,
+                        'clears': 0,
+                        'average': 0,
+                    }
+
+                attempts[songid][songchart]['total'] += remote_attempts[songid][songchart]['plays']
+                attempts[songid][songchart]['clears'] += remote_attempts[songid][songchart]['clears']
+
+        return attempts
+
+    def update_score(
+        self,
+        userid: Optional[UserID],
+        songid: int,
+        chart: int,
+        points: int,
+        clear_type: int,
+        grade: int,
+        combo: int,
+        stats: Optional[Dict[str, int]]=None,
+    ) -> None:
+        """
+        Given various pieces of a score, update the user's high score and score
+        history in a controlled manner, so all games in SDVX series can expect
+        the same attributes in a score.
+        """
+        # Range check clear type
+        if clear_type not in [
+            self.CLEAR_TYPE_NO_PLAY,
+            self.CLEAR_TYPE_FAILED,
+            self.CLEAR_TYPE_CLEAR,
+            self.CLEAR_TYPE_HARD_CLEAR,
+            self.CLEAR_TYPE_ULTIMATE_CHAIN,
+            self.CLEAR_TYPE_PERFECT_ULTIMATE_CHAIN,
+        ]:
+            raise Exception("Invalid clear type value {}".format(clear_type))
+
+        #  Range check grade
+        if grade not in [
+            self.GRADE_NO_PLAY,
+            self.GRADE_D,
+            self.GRADE_C,
+            self.GRADE_B,
+            self.GRADE_A,
+            self.GRADE_A_PLUS,
+            self.GRADE_AA,
+            self.GRADE_AA_PLUS,
+            self.GRADE_AAA,
+            self.GRADE_AAA_PLUS,
+            self.GRADE_S,
+        ]:
+            raise Exception("Invalid clear type value {}".format(grade))
+
+        if userid is not None:
+            oldscore = self.data.local.music.get_score(
+                self.game,
+                self.version,
+                userid,
+                songid,
+                chart,
+            )
+        else:
+            oldscore = None
+
+        # Score history is verbatum, instead of highest score
+        history = ValidatedDict({})
+        oldpoints = points
+
+        if oldscore is None:
+            # If it is a new score, create a new dictionary to add to
+            scoredata = ValidatedDict({})
+            raised = True
+            highscore = True
+        else:
+            # Set the score to any new record achieved
+            raised = points > oldscore.points
+            highscore = points >= oldscore.points
+            points = max(oldscore.points, points)
+            scoredata = oldscore.data
+
+        # Replace clear type and grade
+        scoredata.replace_int('clear_type', max(scoredata.get_int('clear_type'), clear_type))
+        history.replace_int('clear_type', clear_type)
+        scoredata.replace_int('grade', max(scoredata.get_int('grade'), grade))
+        history.replace_int('grade', grade)
+
+        # If we have a combo, replace it
+        scoredata.replace_int('combo', max(scoredata.get_int('combo'), combo))
+        history.replace_int('combo', combo)
+
+        # If we have play stats, replace it
+        if stats is not None:
+            if raised:
+                # We have stats, and there's a new high score, update the stats
+                scoredata.replace_dict('stats', stats)
+            history.replace_dict('stats', stats)
+
+        # Look up where this score was earned
+        lid = self.get_machine_id()
+
+        if userid is not None:
+            # Write the new score back
+            self.data.local.music.put_score(
+                self.game,
+                self.version,
+                userid,
+                songid,
+                chart,
+                lid,
+                points,
+                scoredata,
+                highscore,
+            )
+
+        # Save the history of this score too
+        self.data.local.music.put_attempt(
+            self.game,
+            self.version,
+            userid,
+            songid,
+            chart,
+            lid,
+            oldpoints,
+            history,
+            raised,
+        )

bemani/backend/sdvx/booth.py

@@ -0,0 +1,522 @@
+# vim: set fileencoding=utf-8
+import copy
+from typing import Any, Dict, Optional, Tuple
+
+from bemani.backend.ess import EventLogHandler
+from bemani.backend.sdvx.base import SoundVoltexBase
+from bemani.common import ValidatedDict, VersionConstants, ID, intish
+from bemani.data import Score, UserID
+from bemani.protocol import Node
+
+
+class SoundVoltexBooth(
+    EventLogHandler,
+    SoundVoltexBase,
+):
+
+    name = 'SOUND VOLTEX BOOTH'
+    version = VersionConstants.SDVX_BOOTH
+
+    GAME_LIMITED_LOCKED = 1
+    GAME_LIMITED_UNLOCKED = 2
+
+    GAME_CURRENCY_PACKETS = 0
+    GAME_CURRENCY_BLOCKS = 1
+
+    GAME_CLEAR_TYPE_NO_CLEAR = 1
+    GAME_CLEAR_TYPE_CLEAR = 2
+    GAME_CLEAR_TYPE_ULTIMATE_CHAIN = 3
+    GAME_CLEAR_TYPE_PERFECT_ULTIMATE_CHAIN = 4
+
+    GAME_GRADE_NO_PLAY = 0
+    GAME_GRADE_D = 1
+    GAME_GRADE_C = 2
+    GAME_GRADE_B = 3
+    GAME_GRADE_A = 4
+    GAME_GRADE_AA = 5
+    GAME_GRADE_AAA = 6
+
+    @classmethod
+    def get_settings(cls) -> Dict[str, Any]:
+        """
+        Return all of our front-end modifiably settings.
+        """
+        return {
+            'bools': [
+                {
+                    'name': 'Disable Online Matching',
+                    'tip': 'Disable online matching between games.',
+                    'category': 'game_config',
+                    'setting': 'disable_matching',
+                },
+                {
+                    'name': 'Force Song Unlock',
+                    'tip': 'Force unlock all songs.',
+                    'category': 'game_config',
+                    'setting': 'force_unlock_songs',
+                },
+                {
+                    'name': 'Force Appeal Card Unlock',
+                    'tip': 'Force unlock all appeal cards.',
+                    'category': 'game_config',
+                    'setting': 'force_unlock_cards',
+                },
+            ],
+        }
+
+    def previous_version(self) -> Optional[SoundVoltexBase]:
+        return None
+
+    def __game_to_db_clear_type(self, clear_type: int) -> int:
+        return {
+            self.GAME_CLEAR_TYPE_NO_CLEAR: self.CLEAR_TYPE_FAILED,
+            self.GAME_CLEAR_TYPE_CLEAR: self.CLEAR_TYPE_CLEAR,
+            self.GAME_CLEAR_TYPE_ULTIMATE_CHAIN: self.CLEAR_TYPE_ULTIMATE_CHAIN,
+            self.GAME_CLEAR_TYPE_PERFECT_ULTIMATE_CHAIN: self.CLEAR_TYPE_PERFECT_ULTIMATE_CHAIN,
+        }[clear_type]
+
+    def __db_to_game_clear_type(self, clear_type: int) -> int:
+        return {
+            self.CLEAR_TYPE_NO_PLAY: self.GAME_CLEAR_TYPE_NO_CLEAR,
+            self.CLEAR_TYPE_FAILED: self.GAME_CLEAR_TYPE_NO_CLEAR,
+            self.CLEAR_TYPE_CLEAR: self.GAME_CLEAR_TYPE_CLEAR,
+            self.CLEAR_TYPE_HARD_CLEAR: self.GAME_CLEAR_TYPE_CLEAR,
+            self.CLEAR_TYPE_ULTIMATE_CHAIN: self.GAME_CLEAR_TYPE_ULTIMATE_CHAIN,
+            self.CLEAR_TYPE_PERFECT_ULTIMATE_CHAIN: self.GAME_CLEAR_TYPE_PERFECT_ULTIMATE_CHAIN,
+        }[clear_type]
+
+    def __game_to_db_grade(self, grade: int) -> int:
+        return {
+            self.GAME_GRADE_NO_PLAY: self.GRADE_NO_PLAY,
+            self.GAME_GRADE_D: self.GRADE_D,
+            self.GAME_GRADE_C: self.GRADE_C,
+            self.GAME_GRADE_B: self.GRADE_B,
+            self.GAME_GRADE_A: self.GRADE_A,
+            self.GAME_GRADE_AA: self.GRADE_AA,
+            self.GAME_GRADE_AAA: self.GRADE_AAA,
+        }[grade]
+
+    def __db_to_game_grade(self, grade: int) -> int:
+        return {
+            self.GRADE_NO_PLAY: self.GAME_GRADE_NO_PLAY,
+            self.GRADE_D: self.GAME_GRADE_D,
+            self.GRADE_C: self.GAME_GRADE_C,
+            self.GRADE_B: self.GAME_GRADE_B,
+            self.GRADE_A: self.GAME_GRADE_A,
+            self.GRADE_A_PLUS: self.GAME_GRADE_A,
+            self.GRADE_AA: self.GAME_GRADE_AA,
+            self.GRADE_AA_PLUS: self.GAME_GRADE_AA,
+            self.GRADE_AAA: self.GAME_GRADE_AAA,
+            self.GRADE_AAA_PLUS: self.GAME_GRADE_AAA,
+            self.GRADE_S: self.GAME_GRADE_AAA,
+        }[grade]
+
+    def handle_game_exception_request(self, request: Node) -> Node:
+        return Node.void('game')
+
+    def handle_game_entry_s_request(self, request: Node) -> Node:
+        game = Node.void('game')
+        # This should be created on the fly for a lobby that we're in.
+        game.add_child(Node.u32('entry_id', 1))
+        return game
+
+    def handle_game_lounge_request(self, request: Node) -> Node:
+        game = Node.void('game')
+        # Refresh interval in seconds.
+        game.add_child(Node.u32('interval', 10))
+        return game
+
+    def handle_game_entry_e_request(self, request: Node) -> Node:
+        # Lobby destroy method, eid attribute (u32) should be used
+        # to destroy any open lobbies.
+        return Node.void('game')
+
+    def handle_game_frozen_request(self, request: Node) -> Node:
+        game = Node.void('game')
+        game.set_attribute('result', '0')
+        return game
+
+    def handle_game_shop_request(self, request: Node) -> Node:
+        self.update_machine_name(request.child_value('shopname'))
+
+        # Respond with number of milliseconds until next request
+        game = Node.void('game')
+        game.add_child(Node.u32('nxt_time', 1000 * 5 * 60))
+        return game
+
+    def handle_game_common_request(self, request: Node) -> Node:
+        game = Node.void('game')
+        limited = Node.void('limited')
+        game.add_child(limited)
+
+        game_config = self.get_game_config()
+        if game_config.get_bool('force_unlock_songs'):
+            ids = set()
+            songs = self.data.local.music.get_all_songs(self.game, self.version)
+            for song in songs:
+                if song.data.get_int('limited') == self.GAME_LIMITED_LOCKED:
+                    ids.add(song.id)
+
+            for songid in ids:
+                music = Node.void('music')
+                limited.add_child(music)
+                music.set_attribute('id', str(songid))
+                music.set_attribute('flag', str(self.GAME_LIMITED_UNLOCKED))
+
+        event = Node.void('event')
+        game.add_child(event)
+
+        def enable_event(eid: int) -> None:
+            evt = Node.void('info')
+            event.add_child(evt)
+            evt.set_attribute('id', str(eid))
+
+        if not game_config.get_bool('disable_matching'):
+            enable_event(3)  # Matching enabled
+        enable_event(9)  # Rank Soukuu
+        enable_event(13)  # Year-end bonus
+
+        catalog = Node.void('catalog')
+        game.add_child(catalog)
+        songunlocks = self.data.local.game.get_items(self.game, self.version)
+        for unlock in songunlocks:
+            if unlock.type != 'song_unlock':
+                continue
+
+            info = Node.void('info')
+            catalog.add_child(info)
+            info.set_attribute('id', str(unlock.id))
+            info.set_attribute('currency', str(self.GAME_CURRENCY_BLOCKS))
+            info.set_attribute('price', str(unlock.data.get_int('blocks')))
+
+        kacinfo = Node.void('kacinfo')
+        game.add_child(kacinfo)
+        kacinfo.add_child(Node.u32('note00', 0))
+        kacinfo.add_child(Node.u32('note01', 0))
+        kacinfo.add_child(Node.u32('note02', 0))
+        kacinfo.add_child(Node.u32('note10', 0))
+        kacinfo.add_child(Node.u32('note11', 0))
+        kacinfo.add_child(Node.u32('note12', 0))
+        kacinfo.add_child(Node.u32('rabbeat0', 0))
+        kacinfo.add_child(Node.u32('rabbeat1', 0))
+
+        return game
+
+    def handle_game_hiscore_request(self, request: Node) -> Node:
+        game = Node.void('game')
+
+        # Ranking system I think?
+        for i in range(1, 21):
+            ranking = Node.void('ranking')
+            game.add_child(ranking)
+            ranking.set_attribute('id', str(i))
+
+        hiscore = Node.void('hiscore')
+        game.add_child(hiscore)
+        hiscore.set_attribute('type', '1')
+
+        records = self.data.remote.music.get_all_records(self.game, self.version)
+
+        # Organize by song->chart
+        records_by_id: Dict[int, Dict[int, Tuple[UserID, Score]]] = {}
+        missing_users = []
+        for record in records:
+            userid, score = record
+            if score.id not in records_by_id:
+                records_by_id[score.id] = {}
+
+            records_by_id[score.id][score.chart] = record
+            missing_users.append(userid)
+
+        users = {userid: profile for (userid, profile) in self.get_any_profiles(missing_users)}
+
+        # Output records
+        for songid in records_by_id:
+            music = Node.void('music')
+            hiscore.add_child(music)
+            music.set_attribute('id', str(songid))
+
+            for chart in records_by_id[songid]:
+                note = Node.void('note')
+                music.add_child(note)
+                note.set_attribute('type', str(chart))
+
+                userid, score = records_by_id[songid][chart]
+                note.set_attribute('score', str(score.points))
+                note.set_attribute('name', users[userid].get_str('name'))
+
+        return game
+
+    def handle_game_new_request(self, request: Node) -> Node:
+        refid = request.attribute('refid')
+        name = request.attribute('name')
+        loc = ID.parse_machine_id(request.attribute('locid'))
+        self.new_profile_by_refid(refid, name, loc)
+
+        root = Node.void('game')
+        return root
+
+    def handle_game_load_request(self, request: Node) -> Node:
+        refid = request.attribute('dataid')
+        root = self.get_profile_by_refid(refid)
+        if root is None:
+            root = Node.void('game')
+            root.set_attribute('none', '1')
+        return root
+
+    def handle_game_save_request(self, request: Node) -> Node:
+        refid = request.attribute('refid')
+
+        if refid is not None:
+            userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        else:
+            userid = None
+
+        if userid is not None:
+            oldprofile = self.get_profile(userid)
+            newprofile = self.unformat_profile(userid, request, oldprofile)
+        else:
+            newprofile = None
+
+        if userid is not None and newprofile is not None:
+            self.put_profile(userid, newprofile)
+
+        return Node.void('game')
+
+    def handle_game_load_m_request(self, request: Node) -> Node:
+        refid = request.attribute('dataid')
+
+        if refid is not None:
+            userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        else:
+            userid = None
+
+        if userid is not None:
+            scores = self.data.remote.music.get_scores(self.game, self.version, userid)
+        else:
+            scores = []
+
+        # Organize by song->chart
+        scores_by_id: Dict[int, Dict[int, Score]] = {}
+        for score in scores:
+            if score.id not in scores_by_id:
+                scores_by_id[score.id] = {}
+
+            scores_by_id[score.id][score.chart] = score
+
+        # Output to the game
+        game = Node.void('game')
+        for songid in scores_by_id:
+            music = Node.void('music')
+            game.add_child(music)
+            music.set_attribute('music_id', str(songid))
+
+            for chart in scores_by_id[songid]:
+                typenode = Node.void('type')
+                music.add_child(typenode)
+                typenode.set_attribute('type_id', str(chart))
+
+                score = scores_by_id[songid][chart]
+                typenode.set_attribute('score', str(score.points))
+                typenode.set_attribute('cnt', str(score.plays))
+                typenode.set_attribute('clear_type', str(self.__db_to_game_clear_type(score.data.get_int('clear_type'))))
+                typenode.set_attribute('score_grade', str(self.__db_to_game_grade(score.data.get_int('grade'))))
+
+        return game
+
+    def handle_game_save_m_request(self, request: Node) -> Node:
+        refid = request.attribute('dataid')
+
+        if refid is not None:
+            userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        else:
+            userid = None
+
+        if userid is None:
+            return Node.void('game')
+
+        musicid = int(request.attribute('music_id'))
+        chart = int(request.attribute('music_type'))
+        score = int(request.attribute('score'))
+        combo = int(request.attribute('max_chain'))
+        grade = self.__game_to_db_grade(int(request.attribute('score_grade')))
+        clear_type = self.__game_to_db_clear_type(int(request.attribute('clear_type')))
+
+        # Save the score
+        self.update_score(
+            userid,
+            musicid,
+            chart,
+            score,
+            clear_type,
+            grade,
+            combo,
+        )
+
+        # No response necessary
+        return Node.void('game')
+
+    def handle_game_buy_request(self, request: Node) -> Node:
+        refid = request.attribute('refid')
+
+        if refid is not None:
+            userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        else:
+            userid = None
+
+        if userid is not None:
+            profile = self.get_profile(userid)
+        else:
+            profile = None
+
+        if userid is not None and profile is not None:
+            # Look up packets and blocks
+            packet = profile.get_int('packet')
+            block = profile.get_int('block')
+
+            # Add on any additional we earned this round
+            packet = packet + (request.child_value('earned_gamecoin_packet') or 0)
+            block = block + (request.child_value('earned_gamecoin_block') or 0)
+
+            # Look up the item to get the actual price and currency used
+            item = self.data.local.game.get_item(self.game, self.version, request.child_value('catalog_id'), 'song_unlock')
+            if item is not None:
+                currency_type = request.child_value('currency_type')
+                if currency_type == self.GAME_CURRENCY_PACKETS:
+                    if 'packets' in item:
+                        # This is a valid purchase
+                        newpacket = packet - item.get_int('packets')
+                        if newpacket < 0:
+                            result = 1
+                        else:
+                            packet = newpacket
+                            result = 0
+                    else:
+                        # Bad transaction
+                        result = 1
+                elif currency_type == self.GAME_CURRENCY_BLOCKS:
+                    if 'blocks' in item:
+                        # This is a valid purchase
+                        newblock = block - item.get_int('blocks')
+                        if newblock < 0:
+                            result = 1
+                        else:
+                            block = newblock
+                            result = 0
+                    else:
+                        # Bad transaction
+                        result = 1
+                else:
+                    # Bad currency type
+                    result = 1
+
+                if result == 0:
+                    # Transaction is valid, update the profile with new packets and blocks
+                    profile.replace_int('packet', packet)
+                    profile.replace_int('block', block)
+                    self.put_profile(userid, profile)
+            else:
+                # Bad catalog ID
+                result = 1
+        else:
+            # Unclear what to do here, return a bad response
+            packet = 0
+            block = 0
+            result = 1
+
+        game = Node.void('game')
+        game.add_child(Node.u32('gamecoin_packet', packet))
+        game.add_child(Node.u32('gamecoin_block', block))
+        game.add_child(Node.s8('result', result))
+        return game
+
+    def format_profile(self, userid: UserID, profile: ValidatedDict) -> Node:
+        game = Node.void('game')
+
+        # Generic profile stuff
+        game.add_child(Node.string('name', profile.get_str('name')))
+        game.add_child(Node.string('code', ID.format_extid(profile.get_int('extid'))))
+        game.add_child(Node.u32('gamecoin_packet', profile.get_int('packet')))
+        game.add_child(Node.u32('gamecoin_block', profile.get_int('block')))
+        game.add_child(Node.u32('exp_point', profile.get_int('exp')))
+        game.add_child(Node.u32('m_user_cnt', profile.get_int('m_user_cnt')))
+
+        game_config = self.get_game_config()
+        if game_config.get_bool('force_unlock_cards'):
+            game.add_child(Node.bool_array('have_item', [True] * 512))
+        else:
+            game.add_child(Node.bool_array('have_item', [x > 0 for x in profile.get_int_array('have_item', 512)]))
+        if game_config.get_bool('force_unlock_songs'):
+            game.add_child(Node.bool_array('have_note', [True] * 512))
+        else:
+            game.add_child(Node.bool_array('have_note', [x > 0 for x in profile.get_int_array('have_note', 512)]))
+
+        # Last played stuff
+        lastdict = profile.get_dict('last')
+        last = Node.void('last')
+        game.add_child(last)
+        last.set_attribute('music_id', str(lastdict.get_int('music_id')))
+        last.set_attribute('music_type', str(lastdict.get_int('music_type')))
+        last.set_attribute('sort_type', str(lastdict.get_int('sort_type')))
+        last.set_attribute('headphone', str(lastdict.get_int('headphone')))
+        last.set_attribute('hispeed', str(lastdict.get_int('hispeed')))
+        last.set_attribute('appeal_id', str(lastdict.get_int('appeal_id')))
+        last.set_attribute('frame0', str(lastdict.get_int('frame0')))
+        last.set_attribute('frame1', str(lastdict.get_int('frame1')))
+        last.set_attribute('frame2', str(lastdict.get_int('frame2')))
+        last.set_attribute('frame3', str(lastdict.get_int('frame3')))
+        last.set_attribute('frame4', str(lastdict.get_int('frame4')))
+
+        return game
+
+    def unformat_profile(self, userid: UserID, request: Node, oldprofile: ValidatedDict) -> ValidatedDict:
+        newprofile = copy.deepcopy(oldprofile)
+
+        # Update experience and in-game currencies
+        earned_gamecoin_packet = request.child_value('earned_gamecoin_packet')
+        if earned_gamecoin_packet is not None:
+            newprofile.replace_int('packet', newprofile.get_int('packet') + earned_gamecoin_packet)
+        earned_gamecoin_block = request.child_value('earned_gamecoin_block')
+        if earned_gamecoin_block is not None:
+            newprofile.replace_int('block', newprofile.get_int('block') + earned_gamecoin_block)
+        gain_exp = request.child_value('gain_exp')
+        if gain_exp is not None:
+            newprofile.replace_int('exp', newprofile.get_int('exp') + gain_exp)
+
+        # Miscelaneous stuff
+        newprofile.replace_int('m_user_cnt', request.child_value('m_user_cnt'))
+
+        # Update user's unlock status if we aren't force unlocked
+        game_config = self.get_game_config()
+        if not game_config.get_bool('force_unlock_cards'):
+            have_item = request.child_value('have_item')
+            if have_item is not None:
+                newprofile.replace_int_array('have_item', 512, [1 if x else 0 for x in have_item])
+        if not game_config.get_bool('force_unlock_songs'):
+            have_note = request.child_value('have_note')
+            if have_note is not None:
+                newprofile.replace_int_array('have_note', 512, [1 if x else 0 for x in have_note])
+
+        # Grab last information.
+        lastdict = newprofile.get_dict('last')
+        lastdict.replace_int('headphone', request.child_value('headphone'))
+        lastdict.replace_int('hispeed', request.child_value('hispeed'))
+        lastdict.replace_int('appeal_id', request.child_value('appeal_id'))
+        lastdict.replace_int('frame0', request.child_value('frame0'))
+        lastdict.replace_int('frame1', request.child_value('frame1'))
+        lastdict.replace_int('frame2', request.child_value('frame2'))
+        lastdict.replace_int('frame3', request.child_value('frame3'))
+        lastdict.replace_int('frame4', request.child_value('frame4'))
+        last = request.child('last')
+        if last is not None:
+            lastdict.replace_int('music_id', intish(last.attribute('music_id')))
+            lastdict.replace_int('music_type', intish(last.attribute('music_type')))
+            lastdict.replace_int('sort_type', intish(last.attribute('sort_type')))
+
+        # Save back last information gleaned from results
+        newprofile.replace_dict('last', lastdict)
+
+        # Keep track of play statistics
+        self.update_play_statistics(userid)
+
+        return newprofile

bemani/backend/sdvx/factory.py

@@ -0,0 +1,80 @@
+from typing import Any, Dict, Optional
+
+from bemani.backend.base import Base, Factory
+from bemani.backend.sdvx.booth import SoundVoltexBooth
+from bemani.backend.sdvx.infiniteinfection import SoundVoltexInfiniteInfection
+from bemani.backend.sdvx.gravitywars import SoundVoltexGravityWars
+from bemani.backend.sdvx.gravitywars_s1 import SoundVoltexGravityWarsSeason1
+from bemani.backend.sdvx.gravitywars_s2 import SoundVoltexGravityWarsSeason2
+from bemani.backend.sdvx.heavenlyhaven import SoundVoltexHeavenlyHaven
+from bemani.common import Model, VersionConstants
+from bemani.data import Data
+
+
+class SoundVoltexFactory(Factory):
+
+    MANAGED_CLASSES = [
+        SoundVoltexBooth,
+        SoundVoltexInfiniteInfection,
+        SoundVoltexGravityWars,
+        SoundVoltexHeavenlyHaven,
+    ]
+
+    @classmethod
+    def register_all(cls) -> None:
+        for game in ['KFC']:
+            Base.register(game, SoundVoltexFactory)
+
+    @classmethod
+    def create(cls, data: Data, config: Dict[str, Any], model: Model, parentmodel: Optional[Model]=None) -> Optional[Base]:
+
+        def version_from_date(date: int) -> Optional[int]:
+            if date < 2013060500:
+                return VersionConstants.SDVX_BOOTH
+            elif date >= 2013060500 and date < 2014112000:
+                return VersionConstants.SDVX_INFINITE_INFECTION
+            elif date >= 2014112000 and date < 2016122100:
+                return VersionConstants.SDVX_GRAVITY_WARS
+            elif date >= 2016122100:
+                return VersionConstants.SDVX_HEAVENLY_HAVEN
+            return None
+
+        if model.game == 'KFC':
+            if model.version is None:
+                if parentmodel is None:
+                    return None
+
+                # We have no way to tell apart newer versions. However, we can make
+                # an educated guess if we happen to be summoned for old profile lookup.
+                if parentmodel.game != 'KFC':
+                    return None
+
+                parentversion = version_from_date(parentmodel.version)
+                if parentversion == VersionConstants.SDVX_INFINITE_INFECTION:
+                    return SoundVoltexBooth(data, config, model)
+                if parentversion == VersionConstants.SDVX_GRAVITY_WARS:
+                    return SoundVoltexInfiniteInfection(data, config, model)
+                if parentversion == VersionConstants.SDVX_HEAVENLY_HAVEN:
+                    # We return the generic here because this is usually for profile
+                    # checks, which means we only care about existence.
+                    return SoundVoltexGravityWars(data, config, model)
+
+                # Unknown older version
+                return None
+
+        version = version_from_date(model.version)
+        if version == VersionConstants.SDVX_BOOTH:
+            return SoundVoltexBooth(data, config, model)
+        if version == VersionConstants.SDVX_INFINITE_INFECTION:
+            return SoundVoltexInfiniteInfection(data, config, model)
+        if version == VersionConstants.SDVX_GRAVITY_WARS:
+            # Determine which season
+            if model.version < 2015120400:
+                return SoundVoltexGravityWarsSeason1(data, config, model)
+            else:
+                return SoundVoltexGravityWarsSeason2(data, config, model)
+        if version == VersionConstants.SDVX_HEAVENLY_HAVEN:
+            return SoundVoltexHeavenlyHaven(data, config, model)
+
+        # Unknown game
+        return None

bemani/backend/sdvx/gravitywars.py

@@ -0,0 +1,516 @@
+# vim: set fileencoding=utf-8
+from typing import Any, Dict, List, Optional
+
+from bemani.backend.ess import EventLogHandler
+from bemani.backend.sdvx.base import SoundVoltexBase
+from bemani.backend.sdvx.infiniteinfection import SoundVoltexInfiniteInfection
+from bemani.common import ID, VersionConstants
+from bemani.protocol import Node
+
+
+class SoundVoltexGravityWars(
+    EventLogHandler,
+    SoundVoltexBase,
+):
+
+    name = 'SOUND VOLTEX III GRAVITY WARS'
+    version = VersionConstants.SDVX_GRAVITY_WARS
+
+    GAME_LIMITED_LOCKED = 1
+    GAME_LIMITED_UNLOCKABLE = 2
+    GAME_LIMITED_UNLOCKED = 3
+
+    GAME_CURRENCY_PACKETS = 0
+    GAME_CURRENCY_BLOCKS = 1
+
+    GAME_CLEAR_TYPE_NO_CLEAR = 1
+    GAME_CLEAR_TYPE_CLEAR = 2
+    GAME_CLEAR_TYPE_HARD_CLEAR = 3
+    GAME_CLEAR_TYPE_ULTIMATE_CHAIN = 4
+    GAME_CLEAR_TYPE_PERFECT_ULTIMATE_CHAIN = 5
+
+    GAME_GRADE_NO_PLAY = 0
+    GAME_GRADE_D = 1
+    GAME_GRADE_C = 2
+    GAME_GRADE_B = 3
+    GAME_GRADE_A = 4
+    GAME_GRADE_AA = 5
+    GAME_GRADE_AAA = 6
+
+    GAME_CATALOG_TYPE_SONG = 0
+    GAME_CATALOG_TYPE_APPEAL_CARD = 1
+    GAME_CATALOG_TYPE_CREW = 4
+
+    GAME_GAUGE_TYPE_SKILL = 1
+
+    @classmethod
+    def get_settings(cls) -> Dict[str, Any]:
+        """
+        Return all of our front-end modifiably settings.
+        """
+        return {
+            'bools': [
+                {
+                    'name': 'Disable Online Matching',
+                    'tip': 'Disable online matching between games.',
+                    'category': 'game_config',
+                    'setting': 'disable_matching',
+                },
+                {
+                    'name': 'Force Song Unlock',
+                    'tip': 'Force unlock all songs.',
+                    'category': 'game_config',
+                    'setting': 'force_unlock_songs',
+                },
+                {
+                    'name': 'Force Appeal Card Unlock',
+                    'tip': 'Force unlock all appeal cards.',
+                    'category': 'game_config',
+                    'setting': 'force_unlock_cards',
+                },
+                {
+                    'name': 'Force Crew Card Unlock',
+                    'tip': 'Force unlock all crew and subcrew cards.',
+                    'category': 'game_config',
+                    'setting': 'force_unlock_crew',
+                },
+            ],
+        }
+
+    def previous_version(self) -> Optional[SoundVoltexBase]:
+        return SoundVoltexInfiniteInfection(self.data, self.config, self.model)
+
+    def _get_skill_analyzer_courses(self) -> List[Dict[str, Any]]:
+        # This is overridden in S1/S2 code.
+        return []
+
+    def _get_skill_analyzer_seasons(self) -> Dict[int, str]:
+        # This is overridden in S1/S2 code.
+        return {}
+
+    def _get_extra_events(self) -> List[int]:
+        # This is overridden in S1/S2 code.
+        return []
+
+    def __game_to_db_clear_type(self, clear_type: int) -> int:
+        return {
+            self.GAME_CLEAR_TYPE_NO_CLEAR: self.CLEAR_TYPE_FAILED,
+            self.GAME_CLEAR_TYPE_CLEAR: self.CLEAR_TYPE_CLEAR,
+            self.GAME_CLEAR_TYPE_HARD_CLEAR: self.CLEAR_TYPE_HARD_CLEAR,
+            self.GAME_CLEAR_TYPE_ULTIMATE_CHAIN: self.CLEAR_TYPE_ULTIMATE_CHAIN,
+            self.GAME_CLEAR_TYPE_PERFECT_ULTIMATE_CHAIN: self.CLEAR_TYPE_PERFECT_ULTIMATE_CHAIN,
+        }[clear_type]
+
+    def __db_to_game_clear_type(self, clear_type: int) -> int:
+        return {
+            self.CLEAR_TYPE_NO_PLAY: self.GAME_CLEAR_TYPE_NO_CLEAR,
+            self.CLEAR_TYPE_FAILED: self.GAME_CLEAR_TYPE_NO_CLEAR,
+            self.CLEAR_TYPE_CLEAR: self.GAME_CLEAR_TYPE_CLEAR,
+            self.CLEAR_TYPE_HARD_CLEAR: self.GAME_CLEAR_TYPE_HARD_CLEAR,
+            self.CLEAR_TYPE_ULTIMATE_CHAIN: self.GAME_CLEAR_TYPE_ULTIMATE_CHAIN,
+            self.CLEAR_TYPE_PERFECT_ULTIMATE_CHAIN: self.GAME_CLEAR_TYPE_PERFECT_ULTIMATE_CHAIN,
+        }[clear_type]
+
+    def __game_to_db_grade(self, grade: int) -> int:
+        return {
+            self.GAME_GRADE_NO_PLAY: self.GRADE_NO_PLAY,
+            self.GAME_GRADE_D: self.GRADE_D,
+            self.GAME_GRADE_C: self.GRADE_C,
+            self.GAME_GRADE_B: self.GRADE_B,
+            self.GAME_GRADE_A: self.GRADE_A,
+            self.GAME_GRADE_AA: self.GRADE_AA,
+            self.GAME_GRADE_AAA: self.GRADE_AAA,
+        }[grade]
+
+    def __db_to_game_grade(self, grade: int) -> int:
+        return {
+            self.GRADE_NO_PLAY: self.GAME_GRADE_NO_PLAY,
+            self.GRADE_D: self.GAME_GRADE_D,
+            self.GRADE_C: self.GAME_GRADE_C,
+            self.GRADE_B: self.GAME_GRADE_B,
+            self.GRADE_A: self.GAME_GRADE_A,
+            self.GRADE_A_PLUS: self.GAME_GRADE_A,
+            self.GRADE_AA: self.GAME_GRADE_AA,
+            self.GRADE_AA_PLUS: self.GAME_GRADE_AA,
+            self.GRADE_AAA: self.GAME_GRADE_AAA,
+            self.GRADE_AAA_PLUS: self.GAME_GRADE_AAA,
+            self.GRADE_S: self.GAME_GRADE_AAA,
+        }[grade]
+
+    def __get_skill_analyzer_skill_levels(self) -> Dict[int, str]:
+        return {
+            0: 'Skill LEVEL 01 岳翔',
+            1: 'Skill LEVEL 02 流星',
+            2: 'Skill LEVEL 03 月衝',
+            3: 'Skill LEVEL 04 瞬光',
+            4: 'Skill LEVEL 05 天極',
+            5: 'Skill LEVEL 06 烈風',
+            6: 'Skill LEVEL 07 雷電',
+            7: 'Skill LEVEL 08 麗華',
+            8: 'Skill LEVEL 09 魔騎士',
+            9: 'Skill LEVEL 10 剛力羅',
+            10: 'Skill LEVEL 11 或帝滅斗',
+            11: 'Skill LEVEL ∞(12) 暴龍天',
+        }
+
+    def handle_game_3_common_request(self, request: Node) -> Node:
+        game = Node.void('game_3')
+        limited = Node.void('music_limited')
+        game.add_child(limited)
+
+        # Song unlock config
+        game_config = self.get_game_config()
+        if game_config.get_bool('force_unlock_songs'):
+            ids = set()
+            songs = self.data.local.music.get_all_songs(self.game, self.version)
+            for song in songs:
+                if song.data.get_int('limited') in (self.GAME_LIMITED_LOCKED, self.GAME_LIMITED_UNLOCKABLE):
+                    ids.add((song.id, song.chart))
+
+            for (songid, chart) in ids:
+                info = Node.void('info')
+                limited.add_child(info)
+                info.add_child(Node.s32('music_id', songid))
+                info.add_child(Node.u8('music_type', chart))
+                info.add_child(Node.u8('limited', self.GAME_LIMITED_UNLOCKED))
+
+        # Event config
+        event = Node.void('event')
+        game.add_child(event)
+
+        def enable_event(eid: int) -> None:
+            evt = Node.void('info')
+            event.add_child(evt)
+            evt.add_child(Node.u32('event_id', eid))
+
+        if not game_config.get_bool('disable_matching'):
+            enable_event(1)  # Matching enabled
+        enable_event(2)  # Floor Infection
+        enable_event(3)  # Policy Break
+        enable_event(60)  # BEMANI Summer Diary
+
+        for eventid in self._get_extra_events():
+            enable_event(eventid)
+
+        # Skill Analyzer config
+        skill_course = Node.void('skill_course')
+        game.add_child(skill_course)
+
+        seasons = self._get_skill_analyzer_seasons()
+        skillnames = self.__get_skill_analyzer_skill_levels()
+        courses = self._get_skill_analyzer_courses()
+        max_level: Dict[int, int] = {}
+        for course in courses:
+            max_level[course['level']] = max(course['season_id'], max_level.get(course['level'], -1))
+        for course in courses:
+            info = Node.void('info')
+            skill_course.add_child(info)
+            info.add_child(Node.s16('course_id', course.get('id', course['level'])))
+            info.add_child(Node.s16('level', course['level']))
+            info.add_child(Node.s32('season_id', course['season_id']))
+            info.add_child(Node.string('season_name', seasons[course['season_id']]))
+            info.add_child(Node.bool('season_new_flg', max_level[course['level']] == course['season_id']))
+            info.add_child(Node.string('course_name', course.get('skill_name', skillnames.get(course['level'], ''))))
+            info.add_child(Node.s16('course_type', 0))
+            info.add_child(Node.s16('skill_name_id', course.get('skill_name_id', course['level'])))
+            info.add_child(Node.bool('matching_assist', course['level'] >= 0 and course['level'] <= 6))
+            info.add_child(Node.s16('gauge_type', self.GAME_GAUGE_TYPE_SKILL))
+            info.add_child(Node.s16('paseli_type', 0))
+
+            for trackno, trackdata in enumerate(course['tracks']):
+                track = Node.void('track')
+                info.add_child(track)
+                track.add_child(Node.s16('track_no', trackno))
+                track.add_child(Node.s32('music_id', trackdata['id']))
+                track.add_child(Node.s8('music_type', trackdata['type']))
+
+        return game
+
+    def handle_game_3_exception_request(self, request: Node) -> Node:
+        return Node.void('game_3')
+
+    def handle_game_3_shop_request(self, request: Node) -> Node:
+        self.update_machine_name(request.child_value('shopname'))
+
+        # Respond with number of milliseconds until next request
+        game = Node.void('game_3')
+        game.add_child(Node.u32('nxt_time', 1000 * 5 * 60))
+        return game
+
+    def handle_game_3_lounge_request(self, request: Node) -> Node:
+        game = Node.void('game_3')
+        # Refresh interval in seconds.
+        game.add_child(Node.u32('interval', 10))
+        return game
+
+    def handle_game_3_entry_s_request(self, request: Node) -> Node:
+        game = Node.void('game_3')
+        # This should be created on the fly for a lobby that we're in.
+        game.add_child(Node.u32('entry_id', 1))
+        return game
+
+    def handle_game_3_entry_e_request(self, request: Node) -> Node:
+        # Lobby destroy method, eid node (u32) should be used
+        # to destroy any open lobbies.
+        return Node.void('game_3')
+
+    def handle_game_3_frozen_request(self, request: Node) -> Node:
+        game = Node.void('game_3')
+        game.add_child(Node.u8('result', 0))
+        return game
+
+    def handle_game_3_save_e_request(self, request: Node) -> Node:
+        # This has to do with Policy Break against ReflecBeat and
+        # floor infection, but we don't implement multi-game support so meh.
+        return Node.void('game_3')
+
+    def handle_game_3_play_e_request(self, request: Node) -> Node:
+        return Node.void('game_3')
+
+    def handle_game_3_buy_request(self, request: Node) -> Node:
+        refid = request.child_value('refid')
+
+        if refid is not None:
+            userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        else:
+            userid = None
+
+        if userid is not None:
+            profile = self.get_profile(userid)
+        else:
+            profile = None
+
+        if userid is not None and profile is not None:
+            # Look up packets and blocks
+            packet = profile.get_int('packet')
+            block = profile.get_int('block')
+
+            # Add on any additional we earned this round
+            packet = packet + (request.child_value('earned_gamecoin_packet') or 0)
+            block = block + (request.child_value('earned_gamecoin_block') or 0)
+
+            currency_type = request.child_value('currency_type')
+            price = request.child_value('item/price')
+            if isinstance(price, list):
+                # Sometimes we end up buying more than one item at once
+                price = sum(price)
+
+            if currency_type == self.GAME_CURRENCY_PACKETS:
+                # This is a valid purchase
+                newpacket = packet - price
+                if newpacket < 0:
+                    result = 1
+                else:
+                    packet = newpacket
+                    result = 0
+            elif currency_type == self.GAME_CURRENCY_BLOCKS:
+                # This is a valid purchase
+                newblock = block - price
+                if newblock < 0:
+                    result = 1
+                else:
+                    block = newblock
+                    result = 0
+            else:
+                # Bad currency type
+                result = 1
+
+            if result == 0:
+                # Transaction is valid, update the profile with new packets and blocks
+                profile.replace_int('packet', packet)
+                profile.replace_int('block', block)
+                self.put_profile(userid, profile)
+
+                # If this was a song unlock, we should mark it as unlocked
+                item_type = request.child_value('item/item_type')
+                item_id = request.child_value('item/item_id')
+                param = request.child_value('item/param')
+
+                if not isinstance(item_type, list):
+                    # Sometimes we buy multiple things at once. Make it easier by always assuming this.
+                    item_type = [item_type]
+                    item_id = [item_id]
+                    param = [param]
+
+                for i in range(len(item_type)):
+                    self.data.local.user.put_achievement(
+                        self.game,
+                        self.version,
+                        userid,
+                        item_id[i],
+                        'item_{}'.format(item_type[i]),
+                        {
+                            'param': param[i],
+                        },
+                    )
+
+        else:
+            # Unclear what to do here, return a bad response
+            packet = 0
+            block = 0
+            result = 1
+
+        game = Node.void('game_3')
+        game.add_child(Node.u32('gamecoin_packet', packet))
+        game.add_child(Node.u32('gamecoin_block', block))
+        game.add_child(Node.s8('result', result))
+        return game
+
+    def handle_game_3_new_request(self, request: Node) -> Node:
+        refid = request.child_value('refid')
+        name = request.child_value('name')
+        loc = ID.parse_machine_id(request.child_value('locid'))
+        self.new_profile_by_refid(refid, name, loc)
+
+        root = Node.void('game_3')
+        return root
+
+    def handle_game_3_load_request(self, request: Node) -> Node:
+        refid = request.child_value('refid')
+        root = self.get_profile_by_refid(refid)
+        if root is not None:
+            return root
+
+        # Figure out if this user has an older profile or not
+        userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+
+        if userid is not None:
+            previous_game = self.previous_version()
+        else:
+            previous_game = None
+
+        if previous_game is not None:
+            profile = previous_game.get_profile(userid)
+        else:
+            profile = None
+
+        if profile is not None:
+            root = Node.void('game_3')
+            root.add_child(Node.u8('result', 2))
+            root.add_child(Node.string('name', profile.get_str('name')))
+            return root
+        else:
+            root = Node.void('game_3')
+            root.add_child(Node.u8('result', 1))
+            return root
+
+    def handle_game_3_save_request(self, request: Node) -> Node:
+        refid = request.child_value('refid')
+
+        if refid is not None:
+            userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        else:
+            userid = None
+
+        if userid is not None:
+            oldprofile = self.get_profile(userid)
+            newprofile = self.unformat_profile(userid, request, oldprofile)
+        else:
+            newprofile = None
+
+        if userid is not None and newprofile is not None:
+            self.put_profile(userid, newprofile)
+
+        return Node.void('game_3')
+
+    def handle_game_3_load_m_request(self, request: Node) -> Node:
+        refid = request.child_value('dataid')
+
+        if refid is not None:
+            userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        else:
+            userid = None
+
+        if userid is not None:
+            scores = self.data.remote.music.get_scores(self.game, self.version, userid)
+        else:
+            scores = []
+
+        # Output to the game
+        game = Node.void('game_3')
+        new = Node.void('new')
+        game.add_child(new)
+
+        for score in scores:
+            music = Node.void('music')
+            new.add_child(music)
+            music.add_child(Node.u32('music_id', score.id))
+            music.add_child(Node.u32('music_type', score.chart))
+            music.add_child(Node.u32('score', score.points))
+            music.add_child(Node.u32('cnt', score.plays))
+            music.add_child(Node.u32('clear_type', self.__db_to_game_clear_type(score.data.get_int('clear_type'))))
+            music.add_child(Node.u32('score_grade', self.__db_to_game_grade(score.data.get_int('grade'))))
+            stats = score.data.get_dict('stats')
+            music.add_child(Node.u32('btn_rate', stats.get_int('btn_rate')))
+            music.add_child(Node.u32('long_rate', stats.get_int('long_rate')))
+            music.add_child(Node.u32('vol_rate', stats.get_int('vol_rate')))
+
+        return game
+
+    def handle_game_3_save_m_request(self, request: Node) -> Node:
+        refid = request.child_value('refid')
+
+        if refid is not None:
+            userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        else:
+            userid = None
+
+        # Doesn't matter if userid is None here, that's an anonymous score
+        musicid = request.child_value('music_id')
+        chart = request.child_value('music_type')
+        points = request.child_value('score')
+        combo = request.child_value('max_chain')
+        clear_type = self.__game_to_db_clear_type(request.child_value('clear_type'))
+        grade = self.__game_to_db_grade(request.child_value('score_grade'))
+        stats = {
+            'btn_rate': request.child_value('btn_rate'),
+            'long_rate': request.child_value('long_rate'),
+            'vol_rate': request.child_value('vol_rate'),
+            'critical': request.child_value('critical'),
+            'near': request.child_value('near'),
+            'error': request.child_value('error'),
+        }
+
+        # Save the score
+        self.update_score(
+            userid,
+            musicid,
+            chart,
+            points,
+            clear_type,
+            grade,
+            combo,
+            stats,
+        )
+
+        # Return a blank response
+        return Node.void('game_3')
+
+    def handle_game_3_save_c_request(self, request: Node) -> Node:
+        refid = request.child_value('dataid')
+
+        if refid is not None:
+            userid = self.data.remote.user.from_refid(self.game, self.version, refid)
+        else:
+            userid = None
+
+        if userid is not None:
+            course_id = request.child_value('crsid')
+            clear_type = request.child_value('ct')
+            achievement_rate = request.child_value('ar')
+            season_id = request.child_value('ssnid')
+
+            self.data.local.user.put_achievement(
+                self.game,
+                self.version,
+                userid,
+                (season_id * 100) + course_id,
+                'course',
+                {
+                    'clear_type': clear_type,
+                    'achievement_rate': achievement_rate,
+                },
+            )
+
+        # Return a blank response
+        return Node.void('game_3')