Magic Files for Map Indexing (map.yml file / description.html / preview.png))

TheDog

We humans can easily read the Date+Time stamp and interpret it. It was just an idea.

LaFayette

All good, just trying to explore the idea in case there is something I'm missing.

While a date and timestamp are easier to know if one version is newer than another, who is to know if that has significance? Does the newer version imply it is incompatible, or just has some newer images maybe? What if the XML is simplified in one, and not the other, does that make a difference?

In the lobby, the game version is horribly cut off, so the current value is not very useful. Even if we made the engine aware of map versions and then complain if versions did not align, would that lead to excessive issues for map versions that are actually compatible?

LaFayette

@TheDog , I had my threads crossed. For this version number, it is never displayed to anyone at all. It is just used by the engine to know when to notify a user that their map is out of date. When to trigger that notification is up to the map maker.

LaFayette

There certainly is a discussion to be had about map versioning.

The engine is not designed to handle different map versions, it is designed to assume that they all interoperate and users will be prompted to and always would download the latest. There are lots of questions on how to handle this. Though, that discussion has very little to do with the new 'magic files' that will be required as part of all map repositories going forward.

LaFayette

Lot of progress has been made for the 2.6 game engine to use a 'map.yml' file. Changes are being proposed now with documentation around the file: https://github.com/triplea-game/triplea/pull/8704/commits/fc6a95cd3ba758f9d5645743ea4243ea3a1d99be

Please jump in with commentary on how the documentation could be better or broadly about the map.yml file (this is perhaps the last chance we have to refine the attribute names and anything else in the file).

For context, not only will the 'map.yml' file be used by the game engine, but it'll be used by the maps-server as well. That server will automatically scan map repositories and index the context of the 'map.yml' file. This will remove the need for the 'triplea_maps.yml' file.

LaFayette

Opening this up further, the map server scanning maps will get interesting.

We have a 'map' folder in each map today because of legacy reasons and how the maps used to be packaged. AFAIK this is no longer needed.

I'm thinking to bulk update the existing maps to remove this folder and move all content up by one.

The maps-server will see a lot of action pretty soon and the description of map instead of being in 'triplea-maps.yaml' will instead be found in the map repository in a specially named 'description.html' file. Similar for the preview image being 'preview.png' For all existing maps I think the map-admin team can help ensure those changes happen.

LaFayette

FWIW, the task list of this project is being kept @ https://github.com/triplea-game/triplea/projects/17

On deck:

renaming all map preview images to be 'preview.png'
extracting map descriptions and adding a 'description.html' with existing description to each map
removing the 'map/' folder in each map repository

beelee

@LaFayette hmm...says everything is done. Not that I woulda been of much if any help anyway.

At any rate, Rock On Rock On !!!.png

djabwana

@LaFayette great progress! Can we take this opportunity to make the version be a string so that it can match the version in the map itself instead of being an integer? especially since the UI renders it as a semantic version when it's not. As far as if the map needs to be updated, a simple inequality check should work great and allow map makers to version the map however they see fit.

LaFayette

7 items in done column, 3 in WIP (work in progress), 4 still be started with potentially more to be added in the 'to-be-started' (TODO) column. Still quite a bit to do. Getting it so the game-engine used 'map.yml' file is crossing the line though, it's a major milestone.

Please focus on the 'map.yml' file structure, names, specifics and anything that could be missing or needs to be otherwise specified. This is essentially the LAST chance to give feedback on the map.yml file (not when 2.6 is released, but now). Once we have created one and uploaded it to the 100+ map repositories, it'll be painful to change and probably won't unless it must.

LaFayette

@djabwana The version value in the map (game) XML is deprecated. Even in 2.5 and before it's not used for anything (it is display ony). In 2.6, the GV column has already been removed from the lobby table and the XML attribute for it will be ignored.

The version value in the download description is another entity and exists in 2.5 (in triplea_maps.yaml). It is only used to notify players that they should update.

One issue with the XML version is that there are multiple XMLs to a map, yet the map needs a single version to be able to notify players of a new updated map download.

Can you suggest a further utility for a semver? Can we solve the problem of also explaining semver in a consistent manner and also having it be used in a consistent way (in other words, if we use a semver, how are we sure that everyone understands it and uses it in a consistent way)? IF we go to semver, how would the engine make further use of it?

The version in the map download description is used by the engine, so it is important that it is used in a consistent and predictable way.

djabwana

@LaFayette Gotcha. My point is that map makers should be able to sem ver if they want to instead of being forced into an integer, so that way if they're like me and they want to tag the source repo and use Maven and stuff (even apart from it being removed from the XML) there won't be a mismatch with the "real" version.

People can use sem ver, but if the version is any old string they can do whatever they want. By opening it up, the version could be 1.0-new-units and that would show up in the map list dialog, the lobby, etc. since it would be the real version not just a discarded XML thing.

I think the integer is overly restrictive, but the upside I'm guessing is comparability? so you can say > or < on the versions? But I was proposing we just open it up and do a simple !=, and it's up to the map maker to ensure that makes sense (e.g. not regress back to 1.0 after they released 2.0 - but they can if it turns out 2.0 was no good...)

LaFayette

There are a number of considerations here. First, should the download version and the version of the map be different version values? If so, what is the utility?

Second, map makers should strive to strenously avoid any incompatible changes. A series of problems arise as the engine is just not designed for players to have anything other than the latest of a map. This aspect defeats most of the benefit of a classic semver.

Next, the utility vs complexity trade-off, what is the benefit of a semver compared to the complexities? The complexities include:

having to explain semver
having to know what an incompatible change is (which is nontrivial, should be understood anyways, and incompatible changes should be avoided!)
being able to learn by example, we could have a "wild-west" of version values otherwise

Another aspect is reconciling the many to one relationship between XML versions and map. The map version really needs to be its own thing so the engine knows whether a player has an out-dated version. Having the map version track against multiple XML versions gets complex.

Yet another aspect is version value management. It is bad practice to have version be maintained in multiple places. If the engine were to derive the latest version from say git tags, that would be quite interesting and would avoid having to record the version value anywhere at all. On the other hand, that introduces its own complexities and we are strenuously optimizing for simplicity when it comes to map upload (there is a lot of resistance to git repositories, starting with having to create an account). On the other hand we can flip the perspective and use feature branches to control non-latest versions and then the engine only needs to care about the latest merged into master. The engine could perhaps automatically pick up the last 'sha' of master and record that as a version. Then a downside there is players are notified whenever there is any change to map (which would be okay if using feature branches were a more common thing, but that introduces more learning curve and is therefore avoided)

djabwana

I think we're missing each other here. I'm not at all arguing for sem ver, and I 100% am arguing for not having two different versions. All I'm arguing for is switching it from an int to a String in the YAML.

LaFayette

Sorry, I may be confused. Wouldn't a String be there in order to have a semver? Unless I am totally confused, I realize you are suggesting it be optional for map makers to use semver. Though, I think that creates some complexities that would need to be addressed.

djabwana

I was saying to make it be a string so that map makers can use any versioning system they want, and it would be used by the engine, including but not limited to sem ver. Right now, even assuming the version inside the XML goes away, there are often two different versions: the numeric simple count from triplea_maps.yaml and the "real" version - perhaps a tag, perhaps a thought in the map maker's head (this is after I rebalanced, but before I added cruisers). I'm suggesting that now is the time to open up the version to be an arbitrary string so that it can capture the "real" version, whatever it is. Map makers could literally set it to rebalanced-with-cruiser if they wanted - it wouldn't have to be numeric.

What complexities does that create? The only one is: when a map is out of date and needs to be updated. I'm proposing inequality. So if someone changes their new map.yml to now have rebalanced-with-cruiser-mod-2-with-dans-suggestions, we might suggest they use a new versioning system but the engine just says- this version is now different from the cached one on your system, would you like to update

LaFayette

It is an interesting point that inequality is enough and that the version values do not need to be a POSET (partially ordered set).

I don't think any problems that arise from this are necessarily deal breakers, but in sum they are general consistency problems and it is a goal to keep the learning barrier as minimally low as possible.

The problems I would be concerned with are:

Complexities for new map makers to learn. It's not as simple as "look at another map.yml, see how it is a number, increment that when you want to notify players to download". Instead it becomes nuanced and a person then has to make additional decisions rather than "this is how you do it". It is simpler when there is one way to do something compared to multiple ways.
A version change may not necessitate players to download an update. Perhaps you want your real version value to be 1.0.1, but you actually don't need to bother players with a prompt to update because there are no significant changes. The purpose here drifts, you would not want to update the map.yaml version value unless you want users to update their map, it would not be for tracking the actual version of the map.
Complexity of map update vs game update. That there can be many games to one map. There is an aspect to explain that some map makers like to version their game files (some maps have multiple XML files). This gets complicated to think about compared to a single version for the whole map in map.yml. When there is one XML file, it makes sense to track a global version, but when you have multiple XML files, this is another dimension that can create confusion.
Inconsistency in versioning mechanics, it's confusing if map makers use a common and required element to do the same thing in a different ways. In other words, there is simplicity in unity of purpose and how something is used. It's much simpler to understand "here is what this is, here is how to use it, look at any other map and you will see the same"
Ease of migration. A number of map makers are used to there being a version in map.xml and this confusion even extends to map admins who unnecessarily update this value even though it has no use. If the map.yml supported a semver, it could appear that we have migrated the semver from XML to map.yml. In fact, we have migrated the the version value that was in 'triplea_maps.yaml' (which is an integer), not the version value from XML. Not allowing a semver will help make it clear that this version value is not the deprecated version value that was previously found in XML.

djabwana

@LaFayette Thanks for thinking through this so thoroughly

This is still true- the vast majority of maps would still have a simple number that was incremented (and yaml wouldn't require it to be quoted) and people could use that widespread convention. I know as a professional software developer, I'm somewhat of the minority, but I had the opposite problem- I naively initially put a non-integer in assuming it would work and got a stack trace on deserialization.
I would argue that every version change is worth updating or else it wouldn't be worth a new version. If you want to track incremental progress you can use a feature branch and tag that or something. This does relate to the earlier points about partial download or the assets vs. spec thing you discussed, but conceptually, even a patch version is worth getting because something was fixed or changed
I'm 100% on the same page here. I'm only talking about a top level version in the new file. This is orthogonal IIUC to our discussion here
This is a good point - I think the best one out of the 5
I don't see this as significant- if they think the XML version got "promoted" to the "real" version and players get updates when it changes, that's great! that's exactly what we would want them to think

LaFayette

An ounce of prevention is worth a pound of cure, I'm happy to have the discussion if it leads to a better decision.

re: (3) & (5) the version values on different XML files are independent. If these values were 'promoted', what does it mean when one XML changes but not the other? How can there be a version change if one of the XML did not change? This is why it is kinda wierd to have multple independent game files with one version shared between them.

re: (2) there are some changes that are not worth bothering players with a download. For example removal of commented XML blocks. While I would agree that versioning should track all changes, this version number does have a practical use and we wouldn't want to both players with needless downloads. Consider also the case if map-admins were to bulk change maps for a cosmetic cleanup. If that accompanies a version bump, suddenly every player is prompted to update every map. Furthermore, if the version values are different across the board, then it is actually difficult for map adins.

That is another interesting use-case, let's say (6), if map admins needed to make a change across the board and have everyone update, it could be scripted to bump the version number. If that is ad-hoc per map, that script would be more difficult and it would not necessarily fit and break a map makers versioning scheme when it's suddenly updated to an arbitrary value (like zero).

djabwana

All of this discussion is assuming versions in the XML are going away - like you said it's deprecated. Only the version in map.yaml remains (right?). That's what I mean by "promoted"- all versions scattered across the XMLs have been removed and consolidated and promoted up to the root map.yaml (right?).