Doc: Map Population on website and download list

LaFayette

Description of how map data flows to:

triplea-game client
map listing page
map repositories

Triplea-game client / "Download Maps"-list in TripleA

In short, support server stores a database of all map repositories and the game client sends a request to the support server for a listing of that information. That information then powers the download page.

More detailed, the triplea-support server runs an "indexing" loop where it asks Github for all of the repositories in 'https://github.com/triplea-maps', and then indexes each repository in turn. IIRC it takes a few hours for it to go through all the repositories and to start over. The loop reads all maps (repositories) whether new or old.

The code for the indexing task is at: MapIndexer#L51

With some more details of the different items indexed at: maps/indexing/tasks

The indexer assumes and looks for a couple specific file names:

map.yml
description.html
preview.png

The indexer reads the information from those files and also downloads the map to compute the download size.

Once all that data is gathered, the following database table is updated or has a new row inserted:

 id                  | integer                  |           | not null | nextval('map_index_id_seq'::regclass)
 map_name            | character varying(256)   |           | not null | 
 last_commit_date    | timestamp with time zone |           | not null | 
 repo_url            | character varying(256)   |           | not null | 
 preview_image_url   | character varying(256)   |           | not null | 
 description         | character varying(3000)  |           | not null | 
 download_size_bytes | integer                  |           | not null | 
 download_url        | character varying(256)   |           | not null | 
 default_branch      | character varying(128)   |           | not null | 
 date_created        | timestamp with time zone |           | not null | now()
 date_updated        | timestamp with time zone |           | not null | now()

Map tags are available and are kept in a different database table.

When the game-client opens the download-maps window, the client sends a a request to the server, which then queries the above table and joins that data with map tags. That creates a full data payload that is then returned back to the game-client.

You'll notice the table contains essentially all of the the same data fields that the previous 'triplea_maps.yaml' contained.

How are maps pushed to website? @ https://triplea-game.org/maps-list/maps/ & https://triplea-game.org/map/*specific

There is a "github action" configured here: https://github.com/triplea-game/triplea-game.github.io/blob/main/.github/workflows/update-maps.yml

That runs periodically, queries the maps server for the list of maps, uses a small python script to parse the results, then writes that data to the local file system @ https://github.com/triplea-game/triplea-game.github.io/tree/main/_maps

Then the github action commits any changes and pushes, thereby adding those files to the website repository. Github itself redeploys the website when the repository is updated, thereby updating the website.

So, the flow is:

GH action queries maps server
Python scripts parses the results and creates a single index page lands each map entry as its own file
GH action commits and pushes any file updates
GH itself republishes the website if anything changed

RogerCooper

@LaFayette Is https://github.com/triplea-game/triplea/blob/main/triplea_maps.yaml now deprecated?

What happens if map.yml is duplicated? Some people have work in progress on Github.

I don't much like triplea_maps.yaml but there needs to be some control over the mods.

LaFayette

@RogerCooper If the map_name is duplicated, then we see the example we had with ww2v5 variants being downloaded instead of the correct map.

Note the table has an "ID" and "repo_url", those are both unique. The map-name does not have to be unique though. It should though..

The non-uniqueness of map-name is a design flaw baked in since TripleA was first written. The code treated the name of the map as its unique identity. Ideally the code would use the map IDs wholesale.

Prior to 2.6, the code would look for a map by name, searching for the a map folder with that name, then for an XML with a map_name attribute on it that also had to match. That's updated where the new code ignores the name of the map folder entirely, just looks at "map.yml" to do the matching of the map name. The "map.yml" then indicates where the game XML files are located. Thereby the code is able to load the XML files by using [map_name, game_name] by reading just the 'map.yml' file. Non-uniqueness of map names across all map.yml files is a design flaw.

LaFayette

@RogerCooper Another aspect to duplication of 'map.yaml' - when we fork the repo into 'triplea-maps', (we being the map admins), the map admins should check that 'map.yml' has been updated.

RogerCooper

@LaFayette An error message should go out if there is a non-unique map name in any map.yml

LaFayette

@RogerCooper How, when and where to flag that error message are all open questions. My thoughts on how to solve that - It's on my roadmap to create a HTML status page so we can see the status of map indexing. If a new duplicate map_name is introduced then that status page would reflect the error. I'm not sure if we have to disable the duplicate entirely.

Similarly it's on the roadmap to use IDs as well instead of map names. In which case the map name would appear duplicated in download maps, but the description would be unique and the download would download the right thing.

I'd even like for each map to be stored under its commit-ID on disk as well, then TripleA could finally have a concept of different map versions. Another flaw in TripleA's map handling design is TripleA assumes not only that the map name uniquely identifies a map, but that there is actually just one and only one version of that map.

If TripleA used Map-ID + Commit, TripleA could have multiple versions of a map installed, could use the right one, and could precisely and always download the correct version as needed.

With all that said, the status page is pretty high priority compared to everything, the indexing is an opaque process right now.

RogerCooper

@LaFayette said:
It's on my roadmap to create a HTML status page so we can see the status of map indexing. If a new duplicate map_name is introduced then that status page would reflect the error. I'm not sure if we have to disable the duplicate entirely.

That sounds like a good solution.

If we are changing things, I would like to go back to the old system of having a distinct "mapName". That way, if a mod uses an existing map, there would be no need to copy all the map files over.

LaFayette

@RogerCooper said:

I would like to go back to the old system of having a distinct "mapName".

FWIW, the old system didn't enforce a unique map name. Windows has a case insensitive file system, and because the old system tied the map name to the folder name, you couldn't install two maps with the same name without overwriting. Except, there still was no guarantee that globally across all maps, that the names were unique. Mac and Linux have case sensitive file names, so in the old system you could still have a duplicate name installed so long as the casing was different. Using 'mapName' overall is essentially the design flaw.