Proposing maps.yml index file to be added to each map

[LaFayette]

The proposal is that we add a yml configuration file to each map that will index the games contained in the map bundle. This will be useful for a number of items:

• map uploads
• removes n-layer dependency on map name
• faster map parsing
• allows for thumbnail images to be bundled with the map
• would obsolete the "*.properties" file that gets created on client file systems.

To illustrate, the proposed structure would be the following list:

games:
  - name: game name
    path: <path>/game-name.xml
    game_notes: <path>/game-name-notes.html
    preview_image: <path>/game-name.png

This yml would be in a specifically named file called 'maps.yml' that would live at the top level of any map bundle (map zip or map dir).

Changes to XML

• we would no longer need the map name property
• we would no longer need the game name property

Changes to map loading

The name of a map zip, map repository, and map name in XML would no longer all need to line up. The map zip could have arbitrary names, the name of games would be pulled from the maps.yml file.

Changes to preview image

The preview image would no longer need to be specified in triplea_maps.yaml and could be either an absolute URL path or a relative path. The relative path would be useful for having a preview image be in the zip itself. This goes on to help map uploading as a zip file would be more complete and would simplify that process.

Example

For example, you could create the following map structure:

- maps.yml
- games/
   - world_at_war/
           world_at_war.xml
           world_at_war-notes.html
           world_at_war.png
   - world_at_war_mod/
           world_at_war_mod.xml
           world_at_war_mod-notes.html
           world_at_war_mod.png

In the above example, the names and locations of the files would be arbitrary, they just happen to line up in the example and that perhaps would be a best practice. At the end of the day all of the paths would be found in the maps.yml file. To fully illustrate, the maps.yml file for the above would look like:

games:
  - name: World At War
    path: games/world_at_war/world_at_war.xml
    preview: games/world_at_war/world_at_war.png
    notes: games/world_at_war/world_at_war-notes.html
  - name: World At War Alph Mod
    path: games/world_at_war/world_at_war_mod.xml
    preview: games/world_at_war/world_at_war_mod.png
    notes: games/world_at_war/world_at_war_mod-notes.html

[ff03k64]

you didn't really put a why in your proposal, so not knowing a ton about these things, i can't even ask good questions.

[beelee]

@ff03k64 naming has caused a lot of difficulty over the years. If this makes it easier, would have a valuable impact.

[LaFayette]

@ff03k64 , this list is the 'why' (admittedly it is terse):

@LaFayette said in Proposing maps.yml index file to be added to each map:

map uploads
removes n-layer dependency on map name
faster map parsing
allows for thumbnail images to be bundled with the map
would obsolete the "*.properties" file that gets created on client file systems.

Map Uploads

I'm working on a project where maps can be uploaded from the game itself. This means there will be no interaction by map makers with github repositories. A person opens up the map maker tools: selects a new "upload map" option, selects their map folder or zip, enters their lobby login username and password, and then clicks "upload", and they are done!

This process raises a few problems to solve. Notably when you enter a map into 'triplea_maps.yaml' one is adding the metadata that is intended to be replaced by a 'map.yml' file. For example, when uploading, we need a way to know the name of the map. We could have an upload form to enter this in, but if we do that then you have to enter the same name, each time, and take care to avoid typo's. To some extent it's easier if this information just lives with the map.

To be explicit, map uploads would not only remove all of the map repositories by hosting maps in database instead, it would also remove any need to interact with the 'triplea_maps.yaml' file as well.

removes n-layer dependency on map name

This is a very brittle aspect for how maps are loaded and is a weak design. TripleA has a lot of reliance on "magic" naming, which is known to be generally brittle (easily broken). A recent map upload had this exact problem and we had to debug. Specifically you need to ensure that a map repository is named such that the zip file that comes from it will match the name of the 'map name' attribute in each XML.

There is just no real need for this if we can explicitly determine the name of a map from a configuration file that lives with the map. This removes the need to have a zip file be named in a special way, no need to have a repository with a special name, and no need at all to have a 'mapName' property in the XML. Instead the map name is defined once and would be in the maps.yml.

faster map parsing

Currently map parsing has to scan an entire zip to find all XML files. If instead we can read the 'map.yml' file and then know which files to read from within the zip, map loading should then become more efficient. In part too with explicit paths XML files can be placed in arbitrary locations within the zip file as convenient.

allows for thumbnail images to be bundled with the map

The triplea_maps.yaml defines where thumbnails are to be loaded. This is another step to upload thumbnails somewhere, generally it's easier if the thumbnail can simply be placed within the map zip. When reading a 'map.yml', the zip file to read is the same one containing the 'map.yml', so we can then define a relative path.

Generally though, allowing the thumbnail to live in the zip file keeps the zip much more cohesive and complete, reducing the additional configurations and other files that need to be set up and then referenced in 'triplea_maps.yaml'.

would obsolete the "*.properties" file that gets created on client file systems.

This is primarily a simplification for development and how the engine would work. We are combining here the '.properties' file with a file that lives inside of the zip natively.

[ff03k64]

@LaFayette so besides the work involved, are there any downsides? Will it break current maps? I assume all the current maps would have to be updated with this file. And it sounds like game notes would have to be out in their own file. From what you have said so far, it sounds like a good idea.

[LaFayette]

@ff03k64 - good consideration whether we would have to update maps to have this descriptor file. For maps in the triplea-maps repositories, we could certainly add it. I suspect the engine could fall back to current behavior if the file is missing. Older game engines would still be able to work by defaulting to current behavior as well.

The game notes being in their own file would be optional. If there is no reference to the game notes in the map.yml file then the engine would check the game XMLs for the game notes.

It's still a work in progress for what the map.yml would look like exactly, the latest example is this:

mapName:
previewImage:
games:
   - name:
     path:
     notes:

I'm currently thinking the map description would go into a 'mapDescription.html' file. For context, the map description is what you see when downloading a map, currently today it lives in the triplea_maps.yaml file.

[LaFayette]

Here is an updated version of a potential map.yml file:

map_name:
map_version:
preview_image:
games:
- game_name:
  xml_game_file:
  game_notes_file:
 

[LaFayette]

To explain more how this would work, there are three cases for the engine to handle as it parsing each map-zip in the downloadedMaps folder.

Case 1) map.yml in zip

The engine would first look inside of the zip file for a map.yml file. If it is found, it would read that and then know the paths to each game file, the game name, and path to a game-notes HTML file.

Case 2) map.zip.yml (yml suffix) file

Today, looking at the download maps folder in triplea, we have things like:

world_war_ii_global-master.zip
world_war_ii_global-master.zip.properties

Instead of having a .properties file, we would instead could have a .yml. For example:

world_war_ii_global-master.zip
world_war_ii_global-master.zip.yml

In the above, finding a '.yml' suffix file would be the second best for the engine.

Case 3) No 'map.yml' nor suffix file

Last case if we find a zip that has no map.yml file, and we find no .yml suffix file, then the engine would parse the zip file and generate a .yml suffix file and then use that.

For example if we had:

world_war_ii_global-master.zip

Then in the above, the zip contains no map.yml file, and there is no world_war_ii_global-master.zip.yml file, then the game engine would generate: world_war_ii_global-master.zip.yml and would then use that.

Benefits

Eliminating map-name and file-name dependencies

After this update, we would no longer need a map-name XML tag. The name of the zip file could also be arbitrary as well, the name of the map would come from the .yml file.

Reduce need for a central & single 'triplea_maps.yml' file.

If each map had such a descriptor file, we'd only be missing the map description and we could then have the engine look at all map repositories and automatically generate the list of available maps. This would remove the triplea_maps.yaml file and the engine could instead dynamically generate that file (so no more PR after uploading maps to update triplea_maps.yaml)

[djabwana]

As someone who was bitten by this recently, I 100% agree!
(I'm jdimeo btw from GitHub)

[LaFayette]

An important implication of having a map.yml file to index the games in a map, the engine would no longer scrape the map looking for any XML file to be a game file. Instead it would only use the XML files explicitly referenced from the map.yml file. This would impact map development as you would need to create a map.yml file for the engine to "see" new XML files.

[LaFayette]

Another important implication, having the map.yml file will replace the need for a mapName and gameName attribute to be specified in the XML, both of those attributes will become ignored after the map.yml file is used by the engine (perhaps starting in release 2.6)