Storing Map Index in Database - Deprecating triplea_maps.yaml

LaFayette

I'm working on a few things that I hope map makers will really like. I'm opening this thread to discuss the details while still in design and building out the code.

The initial rev will focus on automating the reading and writing of: https://github.com/triplea-game/triplea/blob/master/triplea_maps.yaml

Currently forecasting this to be a 2.2 or a 2.3 feature.

Naming the maps server

I'm gonna start with the most important : )

The lobby server is currently code-named 'spitfire'. We're going to have a second server for maps, I'm thinking to call it the 'stuka' server.

Github Project

FWIW, this is the github project link where progress can be observed: https://github.com/triplea-game/triplea/projects/17

All dev issues regarding that project can be found with this link: https://github.com/triplea-game/triplea/issues?q=is%3Aopen+is%3Aissue+project%3Atriplea-game%2Ftriplea%2F17

Getting map list from Stuka

The first thing is the game engine will be updated to pull its map list from stuka instead of reading a github file (https://github.com/triplea-game/triplea/blob/master/triplea_maps.yaml). The stuka server would a database to generate the list of maps and return that back to the game client. Essentially this will be exactly the same info as found in the YAML file, but retrieved from database

Pushing update to Stuka Map Index

This is where it gets good. The next step will be to add a UI to triplea-game where a person can open a menu and fill in fields needed for a map version. First you'll enter credentials same you would to log into the lobby server. Then you'll see a UI where you'll be given a list of existing maps that you can update or you can upload a new map. You'll then be given another UI to either publish an update to an existing map or fill in fields for a new map. You can then click 'publish' and your map index will be created in database (no more PRs to update the YAML file).

From then on, anyone downloading maps will see the new or updated map.

For some period of time the server will continue to read the YAML file on github, so we'll still have a legacy route where if you update the YAML file the server will be (eventually) updated. This file will disappear once all clients are migrated to using Stuka.

Considerations

Map Ownership

Map updates will be limited to only those that uploaded them. We'll need to assign owners to each of the maps. This will probably be seeded by the current owners, though most are just the map admins. Anyone who is an 'owner' can update a map.

Sharing Maps

I'm thinking it will be nice to be able to have multiple people be able to push to a map. I'm thinking we can add a UI option for a map owner to add additional people who are allowed to update their map (remember for now it's just the map index information, so not as profound yet as actually just updating the XML, but we'll be getting there )

Map QA & Moderation

An important element here is that new maps can just start appearing on the system. This is potentially bad from a moderation perspective (hopefully nobody will be quite this bored, but bad things can happen). Currently we are gatekeeping maps as we control when a map is updated, when the download URL is updated, and we can see whenever any map contents are changed. This will be relaxed and we won't be able to control when maps are added.

To this extent, we'll probably need a 'moderation' flag on maps, to indicate that maps a 'raw' and could contain anything. I'm not sure if we want to by-default make these available to the public or require that a map-moderator look at the map first and flag it's okay. The latter is a bottleneck and requires resources. We can perhaps label a pretty large group of users as 'trusted' and allow them to do the map-moderation.

Map Versioning

Currently the engine only supports one version of any map, the latest. I'm intending for this to be relaxed and for the engine to be able to manage multiple versions of a map. The different versions will continue to be transparent to the game engine and it'll work only with the latest, but we're laying the foundations early to support multiple map versions.

RoiEX

Some thoughts from my side:
I like the code-names, they add to the feeling of a modern system.

First you'll enter credentials same you would to log into the lobby server.

At this point I'm wondering where servers are separated.
I'm wondering if it would make sense to create a standalone auth server with the single purpose, which is then used by all services that need auth.

Regarding the moderation:
I think having a similar system to the one wikipedia uses for editing articles might be a good move:
Basically anyone can edit or publish a map (editing of course only by "owners" in our case) and the changes are immediately visible on a "not-moderated" changes page, which is opt-in. However until a moderator approves the map, the old listing will be the one that is first displayed to users. WDYT?

LaFayette

Going down the micro-service route could likely hurt us more than it helps. I wouldn't want to break out functionality to that low of a level without additional reason to do so.

@RoiEX the moderation aspect you're suggesting is pretty close to what I'm thinking. The trusted users would have a queue of maps and they'd approve them at which point they would become live. Though, I am nervous about the overhead and complexity that could create. If we lock down the map URLs to github based ones, then we can delay that feature for a little while at least.

Cernel

@LaFayette I'm never sure, due to the traditional terminology being not always followed: By "Map Versioning" do you mean that the maps will have a version or are you referring to the version numbers within the game (xml) files? I'm guessing you are referring to the actual map, and I, then, suppose that the map folder name will change with every version, so to allow to have more than one of the same map on your terminal (similarly to when, in the past, new TripleA instalments were not overwriting any previous ones)?

A thing I always disliked about the current map updating system (beside the time inconsistency due to the time between the map update and the moment in which someone merges your PR to update the "triplea_maps.yaml") is the update versioning number, that it is itself meaningless but for prompting users to update. Will the process be the same (meaning the first time you add a map it will be version 1, the first update will be version 2, and so on)? In the moment that was not really a map versioning number, I'd rather keep track of it by registering the date in which the update was made (maybe limiting to no more than 1 update per day), which would also be a valuable information per se.

If there is a map versioning akin to the release versioning in the current "triplea_maps.yaml" (meaning the first time you add a map it will be version 1, the first update will be version 2, and so on), I think it would be better the version being kept within the map folder (so it will be an actual map versioning) and a versioning increase being strictly required every time anything part of the map is changed (so that, if you change any game file, you have to increase both the game version and the map version, while, if you change only any element outside all game files, you have to increase the map version but not the version of any game).

Just some thoughts, not sure how much on topic, as I'm not really understanding everything that is being said about this new process.

LaFayette

responding to your various questions/concerns @Cernel

due to the traditional terminology being not always followed.

The traditional terminology is frankly wrong and made up, hence the confusion - but that is really besides the point.

By "Map Versioning" do you mean that the maps will have a version or are you referring to the version numbers within the game (xml) files?

The version number in the XML files would go away completely and the entire map would be versioned.

the map folder name will change with every version, so to allow to have more than one of the same map on your terminal (similarly to when, in the past, new TripleA instalments were not overwriting any previous ones)?

TBD, but likely so.

A thing I always disliked about the current map updating system (beside the time inconsistency due to the time between the map update and the moment in which someone merges your PR to update the "triplea_maps.yaml") is the update versioning number, that it is itself meaningless but for prompting users to update.

Yup, the version number in the triplea_maps.yaml is purely for notification purposes. It's much more involved to know the last version that users had and the existing game code has no concept of map versions. (Game versions are also there just for display purposes, that does not even trigger anything, it's only used for display in the lobby 'GV' column).

Will the process be the same (meaning the first time you add a map it will be version 1, the first update will be version 2, and so on)? In the moment that was not really a map versioning number, I'd rather keep track of it by registering the date in which the update was made (maybe limiting to no more than 1 update per day), which would also be a valuable information per se.

When a map is updated would be tracked. A single incrementing sequence for versions is likely. We are getting a bit ahead of ourselves as the game engine does not support multiple map versions. How that is to be done is TBD.

We could easily get into a state where a map truly becomes the XML file and the assets are decoupled from the map. Notably we could have a 'unit editor' feature where unit images, flags, resources, etc are all uploaded individually. Then a game would reference such elements by ID value. When you launch a game you'll have a units folder that has a collection of unit IDs and they are brought in as needed. This avoids all overlap between map resources and allows for map-makers to pick and choose which graphics they would want. Same thing could be done for map polygons. For now though the focus is on deprecating triplea_maps.yaml with a for-like replacement that can be interacted with from in-game.

If there is a map versioning akin to the release versioning in the current "triplea_maps.yaml" (meaning the first time you add a map it will be version 1, the first update will be version 2, and so on), I think it would be better the version being kept within the map folder

I don't think that is readily possible. What if two people update a map at the same time? How do you know a version before you go to upload it? What happens if one version is worked on for a long time and you then do not know the right version number. The version number I think must be external to the map, it should probably not be aware of any other map variants and just worry about being complete in of itself. Any version matching would then be the responsibility of tracking and the game engine.

(so it will be an actual map versioning) and a versioning increase being strictly required every time anything part of the map is changed

Very likely each map upload will be write-once and then read-only (immutable). When uploaded it would get a tracking ID and a version number to know which ones are which and which is latest.

you have to increase both the game version and the map version

That almost certainly will be unnecessary and both versions are just going to be removed from the map-maker domain.

if you change only any element outside all game files, you have to increase the map version but not the version of any game).

A map and its games are going to be more tightly bound. Perhaps even moving to where they are 1:1 and then it becomes a needless distinction.

as I'm not really understanding everything that is being said about this new process.

What is not clear?

RogerCooper

@lafayette Is this going to happen? It seems that we are still using triplea_maps.yaml and awkwardly storing mods on github.

LaFayette

@rogercooper Yes. Though, we're only storing map index information in database, not the map data. The map index information needs to exist regardless of where map data is physically stored. Where to store map data was discussed in some detail.

If we know that maps will be stored in repositories, then there are a number of values that can be automatically computed and no longer would need to be specified at all as index information.

So, what we are doing here is that the lobby server will now automatically scan and find any new map repositories, once about every 10 minutes. After a map is uploaded to repositories, it'll just show up after being scanned. We have in essence done away with the triplea_maps.yaml file, and broken it up and distributed into all of the map repositories as a map.yaml file.

We've gone further than just breaking up the triplea_maps.yaml file, the map.yaml also now serves as a single source of truth for the map name where before the map name had no single source of truth and had to be kept in sync between each game XML file, the actual map folder, the repository name, and triplea_maps.yaml mapName.

To further explain, it's useful to look at the data in triplea_maps.yaml:

- mapName: ...
  mapCategory: ...
  url: https://github.com/triplea-maps/ww2_oil_and_snow_2nd_edition/archive/master.zip
  img: https://github.com/triplea-maps/ww2_oil_and_snow_2nd_edition/raw/master/preview.png
  version: ...
  description: ...

And in comparison, the 'map_name' is the only thing that stays in the 'map.yaml' file.

map_name: ww2_path_to_victory
games:
- {game_name: WW2 Path to Victory, file_name: ww2_path_to_victory.xml}

To explain where the fields went:

mapName: moved to 'map.yaml'
mapCategory: replaced with server side tags maintained by the moderators. So this data is now in database.
url: the server now scans for repositories and so it knows the repository URL. The download URL can be determined if the repository URL is known, so the server can compute this value now and store it in database.
img: if we keep the preview image in a known location in a repository and give it a fixed name, then we can also compute the location of the preview image from the repository URL
version: this was used to detect out of date maps for purposes of available map display in the download window and also used for the out-of-date map scanner that kicks off in the game engine on load once very couple weeks. Otherwise it is not used. To achieve the same solution of knowing when maps are out of date, we can automatically query for the last update time of a repository and compare that to the last update time in file system when a map was last installed or updated.
description: somewhat similar to 'image', if this file is in a well known location within a repository and has a specific file name, then we can compute the location of that file knowing the image URL.

After going through that, all fields except map name are now computed or found at well known locations.

So, long story short, in order to deprecate 'triplea_maps.yaml', we would chop up that file and store it with each individual map and have the server scan repositories to read the 'map.yam' file and then store that information in database.

Cernel

@lafayette To assure consistency, I suggest getting complely rid of the "mapName" and using only the name of the repository, which would be the primary name of the main folder, as well. Otherwise, you will have "map.yaml" map names which are not consistently related to the repository names, and, once you have downloaded the map, you end up having a name of the folder, which is practically the map name, and also a map name defined inside the "map.yaml". I don't even know whether or not the system will assure them being the same or even related. If they are always the same, why to duplicate them, if they are not necessarily the same, what's the point of having one different from the other?

Previously, consistency was virtually assured by the fact that, if you would define a map name (in the game file) which was different from the actual name of the map, the original sking would fail to load.

RogerCooper

@cernel So theoretically I could set up my own repository right now and just have it approved as an authorized repository and added to the scanning process.

Of course, there is nothing to prevent multiple scenarios with the same name.

LaFayette

@Cernel using folder names for the name of the map is not ideal for a number of reasons:

spaces get weird
repository names cannot have spaces in them
it's not expected that the map name to be the folder
a number of characters are restricted from appearing in a folder name
it's not even clear which folder is the 'map folder'. When some maps are extracted you get one folder with files in it, other cases you get a folder containing another folder that then actually contains the map files.
when developing a map, there is no repository, if the name is derived from the folder name then that would have to be kept in sync with repository names; and that can get really complicated and requires map makers to know unnecessary detail about how a repository is packaged into a zip file.

If they are always the same, why to duplicate them, if they are not necessarily the same, what's the point of having one different from the other?

They are not necessarily always the same. Before it was something of a requirement with a few complex caveats and hacks to make the name more human readable, no there is no such dependency nor restriction and a map name can now be much more arbitrary.

LaFayette

@rogercooper Are you referring to the triplea-games organization or a repository under your username?

Of course, there is nothing to prevent multiple scenarios with the same name.

Yes, this was an existing problem and will potentially continue to be. The engines behavior (was not and) is not deterministic if multiple scenarios are installed with the same name.

Cernel

@lafayette So now anyone is free to name the repository/folder however it wants? For example, I can decide my current "270bc_wars" has the map name (in the yaml inside the map folder) "270BC Wars" and the repository/folder name "space-wars"? If not everything is acceptable, are there or will there be guidelines?

Anyways, I'm understanding that now we have the "map name" and the "folder name". The "map name" is inside the "yaml" in the map folder, and nowhere else, whereas the "folder name" is always only the name of the repository (differently from before, when the main folder name was generated from the map name in the "triplea_maps.yaml"). Did I get everything right?

For example, if I currently download "WW2 Oil and Snow 2nd Edition" via TripleA 2.5, what I get is this series of folders:
triplea/downloadedMaps/ww2_oil_and_snow_2nd_edition-master.zip/ww2_oil_and_snow_1st_edition-master/map/games/...
That is, the main folder name (ww2_oil_and_snow_2nd_edition-master.zip) is given by the map name in the "triplea_maps.yaml" (namely, "WW2 Oil and Snow 2nd Edition"), automatically converting all spaces to underscores and all uppercase characters to lowercase, whereas the first sub-folder name is simply the name of the repository (namely, https://github.com/triplea-maps/ww2_oil_and_snow_1st_edition).

I understand, instead, that 2.6 will have one single, non-zipped, folder immediately followed by the "map" folder (Correct?). If so, is the (now single) folder name determined by the repository name, the map name (now given by the "mapName" in the "map.yaml" of the map) or by what else?

LaFayette

@cernel

@lafayette So now anyone is free to name the repository/folder however it wants?

Sure, but we can also all be adults about it and try to keep things easy to manage.

the main folder name was generated from the map name in the "triplea_maps.yaml"

It is was actually more complex. There are just two places where the map name is ever displayed:

the download maps window
https://triplea-game.org/maps-list/maps/

The 'map_name' field in triplea_maps.yaml had just those two uses and nothing further. After download map name came from either XML file or save game file and there is a search algorithm to find folders with a likely matching name.

Now the search algorithm is more precise to look at 'map.yaml' files and to look for a matching map name. The map name in that file also carriers over to replace 'triplea_maps.yaml'

That is, the main folder name (ww2_oil_and_snow_2nd_edition-master.zip) is given by the map name in the "triplea_maps.yaml"

In this example the zip file URL is given by 'triplea_maps.yaml'. The name of the zip file is determined by the repository. 'triplea_map.yaml' mapName plays no part in this or how the download is installed on disk.

I understand, instead, that 2.6 will have one single, non-zipped, folder immediately followed by the "map" folder (Correct?).

The maps will be unzipped, yes. The folder containing the map.yml file is the map folder. I don't know if the game engine is 100% consistent about that though, it wasn't before and may still not be.

If so, is the (now single) folder name determined by the repository name,

The name of the zip file downloaded is determined by the repository. The unzip of that does nothing special and unzips the contents into a similarly named folder.

the map name (now given by the "mapName" in the "map.yaml" of the map) or by what else?

The 'mapName' and 'mapFolder' concept are now much more automatic and a map maker need not
be so aware of it anymore. You no longer need to include 'map name' in XML files.

The engine when it launches a map knows where it came from, it opens the map based on you selecting a scenario
that is listed in the map.yml file. That same file has map name, that value is then injected into save
games as was before. When loading a save game, the map is found by searching for 'map.yaml' files with a matching
map name instead of fuzzy matching folders.

Cernel

@lafayette I suggest obliging (by documenting this requirement) all mapmakers to have a repository name which is exactly equal to the map name (in the map.yaml of the repository) with mandatory changes for problematic characters (like stripping all spaces (I think it is better stripping spaces than turning them into underscores.)).

RogerCooper

@cernel Do the files in the repository need to be kept unzipped or can they be zipped? It is certainly easier to manage a repository with a limited number of zipped files.

LaFayette

@cernel it's a very correctable problem if it were to arise. I don't think it's that big of a worry just yet.

@RogerCooper indeed zipped is easier to manage if the map files do not change.

beelee

@lafayette said in Storing Map Index in Database - Deprecating triplea_maps.yaml:

and have the server scan repositories to read the 'map.yam' file and then store that information in database.

so as long as you have a git repo of said map all is good ?

LaFayette

@beelee Basically. The server will scan for repos in the 'triplea-maps' organization, so the repo needs to be in that org. But yeah, if you fork a repo into the 'triplea-maps' org, it'll be picked up shortly after.

Cernel

@lafayette So it won't be anymore possible to keep (broken or elsewise unworthwhile) maps in the official repository yet not in download list? In particular, I've noticed that "Conquest of the World" is now offered for download (which is fine because I believe the game is at least playable).

LaFayette

@cernel I was tempted to create a 'pull' list of maps that should be pulled. It is easier to instead to remove those maps. We do have categories after all though to mark which maps are still works in progress.