Storing Map Index in Database - Deprecating triplea_maps.yaml

[LaFayette]

@rogercooper Yes. Though, we're only storing map index information in database, not the map data. The map index information needs to exist regardless of where map data is physically stored. Where to store map data was discussed in some detail.

If we know that maps will be stored in repositories, then there are a number of values that can be automatically computed and no longer would need to be specified at all as index information.

So, what we are doing here is that the lobby server will now automatically scan and find any new map repositories, once about every 10 minutes. After a map is uploaded to repositories, it'll just show up after being scanned. We have in essence done away with the triplea_maps.yaml file, and broken it up and distributed into all of the map repositories as a map.yaml file.

We've gone further than just breaking up the triplea_maps.yaml file, the map.yaml also now serves as a single source of truth for the map name where before the map name had no single source of truth and had to be kept in sync between each game XML file, the actual map folder, the repository name, and triplea_maps.yaml mapName.

To further explain, it's useful to look at the data in triplea_maps.yaml:

- mapName: ...
  mapCategory: ...
  url: https://github.com/triplea-maps/ww2_oil_and_snow_2nd_edition/archive/master.zip
  img: https://github.com/triplea-maps/ww2_oil_and_snow_2nd_edition/raw/master/preview.png
  version: ...
  description: ...

And in comparison, the 'map_name' is the only thing that stays in the 'map.yaml' file.

map_name: ww2_path_to_victory
games:
- {game_name: WW2 Path to Victory, file_name: ww2_path_to_victory.xml}

To explain where the fields went:

• mapName: moved to 'map.yaml'
• mapCategory: replaced with server side tags maintained by the moderators. So this data is now in database.
• url: the server now scans for repositories and so it knows the repository URL. The download URL can be determined if the repository URL is known, so the server can compute this value now and store it in database.
• img: if we keep the preview image in a known location in a repository and give it a fixed name, then we can also compute the location of the preview image from the repository URL
• version: this was used to detect out of date maps for purposes of available map display in the download window and also used for the out-of-date map scanner that kicks off in the game engine on load once very couple weeks. Otherwise it is not used. To achieve the same solution of knowing when maps are out of date, we can automatically query for the last update time of a repository and compare that to the last update time in file system when a map was last installed or updated.
• description: somewhat similar to 'image', if this file is in a well known location within a repository and has a specific file name, then we can compute the location of that file knowing the image URL.

After going through that, all fields except map name are now computed or found at well known locations.

So, long story short, in order to deprecate 'triplea_maps.yaml', we would chop up that file and store it with each individual map and have the server scan repositories to read the 'map.yam' file and then store that information in database.

[Cernel]

@lafayette To assure consistency, I suggest getting complely rid of the "mapName" and using only the name of the repository, which would be the primary name of the main folder, as well. Otherwise, you will have "map.yaml" map names which are not consistently related to the repository names, and, once you have downloaded the map, you end up having a name of the folder, which is practically the map name, and also a map name defined inside the "map.yaml". I don't even know whether or not the system will assure them being the same or even related. If they are always the same, why to duplicate them, if they are not necessarily the same, what's the point of having one different from the other?

Previously, consistency was virtually assured by the fact that, if you would define a map name (in the game file) which was different from the actual name of the map, the original sking would fail to load.

[RogerCooper]

@cernel So theoretically I could set up my own repository right now and just have it approved as an authorized repository and added to the scanning process.

Of course, there is nothing to prevent multiple scenarios with the same name.

[LaFayette]

@Cernel using folder names for the name of the map is not ideal for a number of reasons:

• spaces get weird
• repository names cannot have spaces in them
• it's not expected that the map name to be the folder
• a number of characters are restricted from appearing in a folder name
• it's not even clear which folder is the 'map folder'. When some maps are extracted you get one folder with files in it, other cases you get a folder containing another folder that then actually contains the map files.
• when developing a map, there is no repository, if the name is derived from the folder name then that would have to be kept in sync with repository names; and that can get really complicated and requires map makers to know unnecessary detail about how a repository is packaged into a zip file.

If they are always the same, why to duplicate them, if they are not necessarily the same, what's the point of having one different from the other?

They are not necessarily always the same. Before it was something of a requirement with a few complex caveats and hacks to make the name more human readable, no there is no such dependency nor restriction and a map name can now be much more arbitrary.

[LaFayette]

@rogercooper Are you referring to the triplea-games organization or a repository under your username?

Of course, there is nothing to prevent multiple scenarios with the same name.

Yes, this was an existing problem and will potentially continue to be. The engines behavior (was not and) is not deterministic if multiple scenarios are installed with the same name.

[Cernel]

@lafayette So now anyone is free to name the repository/folder however it wants? For example, I can decide my current "270bc_wars" has the map name (in the yaml inside the map folder) "270BC Wars" and the repository/folder name "space-wars"? If not everything is acceptable, are there or will there be guidelines?

Anyways, I'm understanding that now we have the "map name" and the "folder name". The "map name" is inside the "yaml" in the map folder, and nowhere else, whereas the "folder name" is always only the name of the repository (differently from before, when the main folder name was generated from the map name in the "triplea_maps.yaml"). Did I get everything right?

For example, if I currently download "WW2 Oil and Snow 2nd Edition" via TripleA 2.5, what I get is this series of folders:
triplea/downloadedMaps/ww2_oil_and_snow_2nd_edition-master.zip/ww2_oil_and_snow_1st_edition-master/map/games/...
That is, the main folder name (ww2_oil_and_snow_2nd_edition-master.zip) is given by the map name in the "triplea_maps.yaml" (namely, "WW2 Oil and Snow 2nd Edition"), automatically converting all spaces to underscores and all uppercase characters to lowercase, whereas the first sub-folder name is simply the name of the repository (namely, https://github.com/triplea-maps/ww2_oil_and_snow_1st_edition).

I understand, instead, that 2.6 will have one single, non-zipped, folder immediately followed by the "map" folder (Correct?). If so, is the (now single) folder name determined by the repository name, the map name (now given by the "mapName" in the "map.yaml" of the map) or by what else?

[LaFayette]

@cernel

@lafayette So now anyone is free to name the repository/folder however it wants?

Sure, but we can also all be adults about it and try to keep things easy to manage.

the main folder name was generated from the map name in the "triplea_maps.yaml"

It is was actually more complex. There are just two places where the map name is ever displayed:

• the download maps window
• https://triplea-game.org/maps-list/maps/

The 'map_name' field in triplea_maps.yaml had just those two uses and nothing further. After download map name came from either XML file or save game file and there is a search algorithm to find folders with a likely matching name.

Now the search algorithm is more precise to look at 'map.yaml' files and to look for a matching map name. The map name in that file also carriers over to replace 'triplea_maps.yaml'

That is, the main folder name (ww2_oil_and_snow_2nd_edition-master.zip) is given by the map name in the "triplea_maps.yaml"

In this example the zip file URL is given by 'triplea_maps.yaml'. The name of the zip file is determined by the repository. 'triplea_map.yaml' mapName plays no part in this or how the download is installed on disk.

I understand, instead, that 2.6 will have one single, non-zipped, folder immediately followed by the "map" folder (Correct?).

The maps will be unzipped, yes. The folder containing the map.yml file is the map folder. I don't know if the game engine is 100% consistent about that though, it wasn't before and may still not be.

If so, is the (now single) folder name determined by the repository name,

The name of the zip file downloaded is determined by the repository. The unzip of that does nothing special and unzips the contents into a similarly named folder.

the map name (now given by the "mapName" in the "map.yaml" of the map) or by what else?

The 'mapName' and 'mapFolder' concept are now much more automatic and a map maker need not
be so aware of it anymore. You no longer need to include 'map name' in XML files.

The engine when it launches a map knows where it came from, it opens the map based on you selecting a scenario
that is listed in the map.yml file. That same file has map name, that value is then injected into save
games as was before. When loading a save game, the map is found by searching for 'map.yaml' files with a matching
map name instead of fuzzy matching folders.

[Cernel]

@lafayette I suggest obliging (by documenting this requirement) all mapmakers to have a repository name which is exactly equal to the map name (in the map.yaml of the repository) with mandatory changes for problematic characters (like stripping all spaces (I think it is better stripping spaces than turning them into underscores.)).

[RogerCooper]

@cernel Do the files in the repository need to be kept unzipped or can they be zipped? It is certainly easier to manage a repository with a limited number of zipped files.

[LaFayette]

@cernel it's a very correctable problem if it were to arise. I don't think it's that big of a worry just yet.

@RogerCooper indeed zipped is easier to manage if the map files do not change.

[beelee]

@lafayette said in Storing Map Index in Database - Deprecating triplea_maps.yaml:

and have the server scan repositories to read the 'map.yam' file and then store that information in database.

so as long as you have a git repo of said map all is good ?

[LaFayette]

@beelee Basically. The server will scan for repos in the 'triplea-maps' organization, so the repo needs to be in that org. But yeah, if you fork a repo into the 'triplea-maps' org, it'll be picked up shortly after.

[Cernel]

@lafayette So it won't be anymore possible to keep (broken or elsewise unworthwhile) maps in the official repository yet not in download list? In particular, I've noticed that "Conquest of the World" is now offered for download (which is fine because I believe the game is at least playable).

[LaFayette]

@cernel I was tempted to create a 'pull' list of maps that should be pulled. It is easier to instead to remove those maps. We do have categories after all though to mark which maps are still works in progress.