Pagination in database get_all_records and get_all_scores

[Subject38]

In jubeat and a few other games (GFDM comes to mind) pagination is requested for the response. In jubeat, get_mdata is called 3 times and a valid response is to send 3 pages of records rather than 1 page of every record in the response. In the gfdm v games, the score request method only allows up to 250 entries per page, and keeps requesting until the server says it's done. This is a request for anyone willing to take it on to implement pagination in get_all_records and get_all_scores that grabs the first x number of entries by music ID. For example, grab all scores between music id 0 and 250, 250 and 500, etc...

[DragonMinded]

So that's what the three fetches in Jubeat are for. Have you confirmed that sending some scores in each results in all of them appearing on the playfield together? Does Jubeat have a limit to number songs it will process per-packet? GFDM pagination makes a lot more sense IMO, its a lot easier to do offset/limit with arbitrary sizes in MySQL than it is to figure out how to evenly divide results into three responses.

IIRC some of the mysql fetch operations already support limited pagination for the frontend. Getting this working correctly with remote server linking might be a bit difficult, however. Probably the best thing to do is look up all the scores for a game when servicing the first packet, cache them somewhere and pull from that until it is empty which would work for both local and remote fetches as well as mean that we have the ability to evenly divide the response for Jubeat. We would need a backend cache for starters as the current cache system is frontend-only.

[DragonMinded]

So, revisiting this now that we have a caching layer for the backend and Jubeat makes use of this to split scores across the three separate music requests. For GFDM, similar logic can be implemented where the initial fetch hydrates the cache and then we pull X songs per request given the client's request for a maximum. See Jubeat for an example of how this would work, but in effect it means we don't need to implement DB or BEMAPI pagination at all, we can just use the cache layer to handle it with a single fetch at the beginning.