Magic Files for Map Indexing (map.yml file / description.html / preview.png))

LaFayette

The category information needs to live outside of the map entirely. If the category changes we should not need to change the map contents and that would require users to re-download the map. I'm thinking the category would live in the maps-server database. The maps-server can query github to get a list of all repositories, from there it can then scrape the 'map.yml' file in each one and create a listing of available maps. This database will replace the main triplea_maps.yaml file. Then the game client when requesting the list of available maps would send a request to the server which would then query it's database to return results.

Partial downloads is not on the radar right now. It'll be quite complex. I'd rather we first do other projects to reduce the size of the download. Like specifically I'd like to see maps broken up and the XML distributed apart from the assets. This would essentially create a pool of assets such as images, maps, and XMLs would be independent and reference the assets by ID. When downloading an XML, the engine would figure out which asset IDs are required and download those. I don't know if we'll ever get to that model, so discussing it now is perhaps premature, but that's more or less the long-term radar on where maps could go.

TheDog

Could the version number of the map be automatically calculated?
Something like;
Date+Time stamp YYYY-MM-DD---HH:MM:SS

So the person uploading the map does not have to worry about the version?

This also has the advantage, it shows when the map was last changed/updated/fixed.

LaFayette

If so, then what would the version be used for @TheDog ?

There is a commit 'sha' attached to any change, every version of a map is already uniquely identified by that.

TheDog

We humans can easily read the Date+Time stamp and interpret it. It was just an idea.

LaFayette

All good, just trying to explore the idea in case there is something I'm missing.

While a date and timestamp are easier to know if one version is newer than another, who is to know if that has significance? Does the newer version imply it is incompatible, or just has some newer images maybe? What if the XML is simplified in one, and not the other, does that make a difference?

In the lobby, the game version is horribly cut off, so the current value is not very useful. Even if we made the engine aware of map versions and then complain if versions did not align, would that lead to excessive issues for map versions that are actually compatible?

LaFayette

@TheDog , I had my threads crossed. For this version number, it is never displayed to anyone at all. It is just used by the engine to know when to notify a user that their map is out of date. When to trigger that notification is up to the map maker.

LaFayette

There certainly is a discussion to be had about map versioning.

The engine is not designed to handle different map versions, it is designed to assume that they all interoperate and users will be prompted to and always would download the latest. There are lots of questions on how to handle this. Though, that discussion has very little to do with the new 'magic files' that will be required as part of all map repositories going forward.

LaFayette

Lot of progress has been made for the 2.6 game engine to use a 'map.yml' file. Changes are being proposed now with documentation around the file: https://github.com/triplea-game/triplea/pull/8704/commits/fc6a95cd3ba758f9d5645743ea4243ea3a1d99be

Please jump in with commentary on how the documentation could be better or broadly about the map.yml file (this is perhaps the last chance we have to refine the attribute names and anything else in the file).

For context, not only will the 'map.yml' file be used by the game engine, but it'll be used by the maps-server as well. That server will automatically scan map repositories and index the context of the 'map.yml' file. This will remove the need for the 'triplea_maps.yml' file.

LaFayette

Opening this up further, the map server scanning maps will get interesting.

We have a 'map' folder in each map today because of legacy reasons and how the maps used to be packaged. AFAIK this is no longer needed.

I'm thinking to bulk update the existing maps to remove this folder and move all content up by one.

The maps-server will see a lot of action pretty soon and the description of map instead of being in 'triplea-maps.yaml' will instead be found in the map repository in a specially named 'description.html' file. Similar for the preview image being 'preview.png' For all existing maps I think the map-admin team can help ensure those changes happen.

LaFayette

FWIW, the task list of this project is being kept @ https://github.com/triplea-game/triplea/projects/17

On deck:

renaming all map preview images to be 'preview.png'
extracting map descriptions and adding a 'description.html' with existing description to each map
removing the 'map/' folder in each map repository

beelee

@LaFayette hmm...says everything is done. Not that I woulda been of much if any help anyway.

At any rate, Rock On Rock On !!!.png

djabwana

@LaFayette great progress! Can we take this opportunity to make the version be a string so that it can match the version in the map itself instead of being an integer? especially since the UI renders it as a semantic version when it's not. As far as if the map needs to be updated, a simple inequality check should work great and allow map makers to version the map however they see fit.

LaFayette

7 items in done column, 3 in WIP (work in progress), 4 still be started with potentially more to be added in the 'to-be-started' (TODO) column. Still quite a bit to do. Getting it so the game-engine used 'map.yml' file is crossing the line though, it's a major milestone.

Please focus on the 'map.yml' file structure, names, specifics and anything that could be missing or needs to be otherwise specified. This is essentially the LAST chance to give feedback on the map.yml file (not when 2.6 is released, but now). Once we have created one and uploaded it to the 100+ map repositories, it'll be painful to change and probably won't unless it must.

LaFayette

@djabwana The version value in the map (game) XML is deprecated. Even in 2.5 and before it's not used for anything (it is display ony). In 2.6, the GV column has already been removed from the lobby table and the XML attribute for it will be ignored.

The version value in the download description is another entity and exists in 2.5 (in triplea_maps.yaml). It is only used to notify players that they should update.

One issue with the XML version is that there are multiple XMLs to a map, yet the map needs a single version to be able to notify players of a new updated map download.

Can you suggest a further utility for a semver? Can we solve the problem of also explaining semver in a consistent manner and also having it be used in a consistent way (in other words, if we use a semver, how are we sure that everyone understands it and uses it in a consistent way)? IF we go to semver, how would the engine make further use of it?

The version in the map download description is used by the engine, so it is important that it is used in a consistent and predictable way.

djabwana

@LaFayette Gotcha. My point is that map makers should be able to sem ver if they want to instead of being forced into an integer, so that way if they're like me and they want to tag the source repo and use Maven and stuff (even apart from it being removed from the XML) there won't be a mismatch with the "real" version.

People can use sem ver, but if the version is any old string they can do whatever they want. By opening it up, the version could be 1.0-new-units and that would show up in the map list dialog, the lobby, etc. since it would be the real version not just a discarded XML thing.

I think the integer is overly restrictive, but the upside I'm guessing is comparability? so you can say > or < on the versions? But I was proposing we just open it up and do a simple !=, and it's up to the map maker to ensure that makes sense (e.g. not regress back to 1.0 after they released 2.0 - but they can if it turns out 2.0 was no good...)

LaFayette

There are a number of considerations here. First, should the download version and the version of the map be different version values? If so, what is the utility?

Second, map makers should strive to strenously avoid any incompatible changes. A series of problems arise as the engine is just not designed for players to have anything other than the latest of a map. This aspect defeats most of the benefit of a classic semver.

Next, the utility vs complexity trade-off, what is the benefit of a semver compared to the complexities? The complexities include:

having to explain semver
having to know what an incompatible change is (which is nontrivial, should be understood anyways, and incompatible changes should be avoided!)
being able to learn by example, we could have a "wild-west" of version values otherwise

Another aspect is reconciling the many to one relationship between XML versions and map. The map version really needs to be its own thing so the engine knows whether a player has an out-dated version. Having the map version track against multiple XML versions gets complex.

Yet another aspect is version value management. It is bad practice to have version be maintained in multiple places. If the engine were to derive the latest version from say git tags, that would be quite interesting and would avoid having to record the version value anywhere at all. On the other hand, that introduces its own complexities and we are strenuously optimizing for simplicity when it comes to map upload (there is a lot of resistance to git repositories, starting with having to create an account). On the other hand we can flip the perspective and use feature branches to control non-latest versions and then the engine only needs to care about the latest merged into master. The engine could perhaps automatically pick up the last 'sha' of master and record that as a version. Then a downside there is players are notified whenever there is any change to map (which would be okay if using feature branches were a more common thing, but that introduces more learning curve and is therefore avoided)

djabwana

I think we're missing each other here. I'm not at all arguing for sem ver, and I 100% am arguing for not having two different versions. All I'm arguing for is switching it from an int to a String in the YAML.

LaFayette

Sorry, I may be confused. Wouldn't a String be there in order to have a semver? Unless I am totally confused, I realize you are suggesting it be optional for map makers to use semver. Though, I think that creates some complexities that would need to be addressed.

djabwana

I was saying to make it be a string so that map makers can use any versioning system they want, and it would be used by the engine, including but not limited to sem ver. Right now, even assuming the version inside the XML goes away, there are often two different versions: the numeric simple count from triplea_maps.yaml and the "real" version - perhaps a tag, perhaps a thought in the map maker's head (this is after I rebalanced, but before I added cruisers). I'm suggesting that now is the time to open up the version to be an arbitrary string so that it can capture the "real" version, whatever it is. Map makers could literally set it to rebalanced-with-cruiser if they wanted - it wouldn't have to be numeric.

What complexities does that create? The only one is: when a map is out of date and needs to be updated. I'm proposing inequality. So if someone changes their new map.yml to now have rebalanced-with-cruiser-mod-2-with-dans-suggestions, we might suggest they use a new versioning system but the engine just says- this version is now different from the cached one on your system, would you like to update

LaFayette

It is an interesting point that inequality is enough and that the version values do not need to be a POSET (partially ordered set).

I don't think any problems that arise from this are necessarily deal breakers, but in sum they are general consistency problems and it is a goal to keep the learning barrier as minimally low as possible.

The problems I would be concerned with are:

Complexities for new map makers to learn. It's not as simple as "look at another map.yml, see how it is a number, increment that when you want to notify players to download". Instead it becomes nuanced and a person then has to make additional decisions rather than "this is how you do it". It is simpler when there is one way to do something compared to multiple ways.
A version change may not necessitate players to download an update. Perhaps you want your real version value to be 1.0.1, but you actually don't need to bother players with a prompt to update because there are no significant changes. The purpose here drifts, you would not want to update the map.yaml version value unless you want users to update their map, it would not be for tracking the actual version of the map.
Complexity of map update vs game update. That there can be many games to one map. There is an aspect to explain that some map makers like to version their game files (some maps have multiple XML files). This gets complicated to think about compared to a single version for the whole map in map.yml. When there is one XML file, it makes sense to track a global version, but when you have multiple XML files, this is another dimension that can create confusion.
Inconsistency in versioning mechanics, it's confusing if map makers use a common and required element to do the same thing in a different ways. In other words, there is simplicity in unity of purpose and how something is used. It's much simpler to understand "here is what this is, here is how to use it, look at any other map and you will see the same"
Ease of migration. A number of map makers are used to there being a version in map.xml and this confusion even extends to map admins who unnecessarily update this value even though it has no use. If the map.yml supported a semver, it could appear that we have migrated the semver from XML to map.yml. In fact, we have migrated the the version value that was in 'triplea_maps.yaml' (which is an integer), not the version value from XML. Not allowing a semver will help make it clear that this version value is not the deprecated version value that was previously found in XML.