Magic Files for Map Indexing (map.yml file / description.html / preview.png))
-
I'm working on a way to avoid having to update: triplea_maps..yaml by instead embedding the needed information in the map bundle itself.
We still need to supply the same information, but in an alternative location. Today the map folder structure at a top level is just:
- map/The data needed from each map is:
- map name
- version (download version, used to notify users of updates)
- preview image location (the link to this image is in
triplea_maps.yaml, typically the image is stored in the map repository) - map description (currently this lives in
triplea_maps.yaml, usually as a CDATA block.
The map name and version will be found in
map.ymlfile. The structure of that file is described in the map.yml forum thread. The description and preview I think can just be files. That would make the top level of a map look like this:- map/ - map.yml ## contains 'map-name' + 'version' - preview.png ## this will be the preview image - description.html ## contains the descriptionAnd there you have it, once those files are added, the engine would be able to pull information automatically without needing an entry in
triplea_maps.yml.I hope for this feature to be live for 2.6, until we get users migrated to 2.6 the triplea_maps.yml file will need an entry. Meanwhile, we can certainly add the above files and 2.5 or less will ignore them and 2.6 or greater will pick them up.
Please provide any feedback that comes up, I'm particularly interested in the file naming and if this all seems easy enough.
-
@LaFayette currently the yaml file also has a category. That could maybe go in the map.yml file?
For future proofing, we could maybe allow multiple categories but only use the first one in a list for now, or just look for certain categories.Something else that i have been thinking about with different map versions is that it would be nice to only download files that have changed instead of a whole map. Could this use the map version number as a kind of benchmark, and just download the files that are different between those versions? In my limited programming experience, that seems like something that would use some of the same files, so it might involve less work if it were to be done at the same time. I don't know if it is worth doing, but downloading the warcraft map in its entirety takes a while now (i think it is 500mb) and is kind of admitting if it is just for a few xml edits.
-
The category information needs to live outside of the map entirely. If the category changes we should not need to change the map contents and that would require users to re-download the map. I'm thinking the category would live in the maps-server database. The maps-server can query github to get a list of all repositories, from there it can then scrape the 'map.yml' file in each one and create a listing of available maps. This database will replace the main
triplea_maps.yamlfile. Then the game client when requesting the list of available maps would send a request to the server which would then query it's database to return results.Partial downloads is not on the radar right now. It'll be quite complex. I'd rather we first do other projects to reduce the size of the download. Like specifically I'd like to see maps broken up and the XML distributed apart from the assets. This would essentially create a pool of assets such as images, maps, and XMLs would be independent and reference the assets by ID. When downloading an XML, the engine would figure out which asset IDs are required and download those. I don't know if we'll ever get to that model, so discussing it now is perhaps premature, but that's more or less the long-term radar on where maps could go.
-
Could the version number of the map be automatically calculated?
Something like;
Date+Time stamp YYYY-MM-DD---HH:MM:SSSo the person uploading the map does not have to worry about the version?
This also has the advantage, it shows when the map was last changed/updated/fixed.
-
If so, then what would the version be used for @TheDog ?
There is a commit 'sha' attached to any change, every version of a map is already uniquely identified by that.
-
We humans can easily read the Date+Time stamp and interpret it. It was just an idea.
-
All good, just trying to explore the idea in case there is something I'm missing.
While a date and timestamp are easier to know if one version is newer than another, who is to know if that has significance? Does the newer version imply it is incompatible, or just has some newer images maybe? What if the XML is simplified in one, and not the other, does that make a difference?
In the lobby, the game version is horribly cut off, so the current value is not very useful. Even if we made the engine aware of map versions and then complain if versions did not align, would that lead to excessive issues for map versions that are actually compatible?
-
@TheDog , I had my threads crossed. For this version number, it is never displayed to anyone at all. It is just used by the engine to know when to notify a user that their map is out of date. When to trigger that notification is up to the map maker.
-
There certainly is a discussion to be had about map versioning.
The engine is not designed to handle different map versions, it is designed to assume that they all interoperate and users will be prompted to and always would download the latest. There are lots of questions on how to handle this. Though, that discussion has very little to do with the new 'magic files' that will be required as part of all map repositories going forward.
-
Lot of progress has been made for the 2.6 game engine to use a 'map.yml' file. Changes are being proposed now with documentation around the file: https://github.com/triplea-game/triplea/pull/8704/commits/fc6a95cd3ba758f9d5645743ea4243ea3a1d99be
Please jump in with commentary on how the documentation could be better or broadly about the map.yml file (this is perhaps the last chance we have to refine the attribute names and anything else in the file).
For context, not only will the 'map.yml' file be used by the game engine, but it'll be used by the maps-server as well. That server will automatically scan map repositories and index the context of the 'map.yml' file. This will remove the need for the 'triplea_maps.yml' file.
closed Read & Generate Map Description YAML (map.yml) files #8704
-
Opening this up further, the map server scanning maps will get interesting.
We have a 'map' folder in each map today because of legacy reasons and how the maps used to be packaged. AFAIK this is no longer needed.
I'm thinking to bulk update the existing maps to remove this folder and move all content up by one.
The maps-server will see a lot of action pretty soon and the description of map instead of being in 'triplea-maps.yaml' will instead be found in the map repository in a specially named 'description.html' file. Similar for the preview image being 'preview.png' For all existing maps I think the map-admin team can help ensure those changes happen.
-
FWIW, the task list of this project is being kept @ https://github.com/triplea-game/triplea/projects/17
On deck:
- renaming all map preview images to be 'preview.png'
- extracting map descriptions and adding a 'description.html' with existing description to each map
- removing the 'map/' folder in each map repository
-
@LaFayette hmm...says everything is done. Not that I woulda been of much if any help anyway.
At any rate, Rock On
-
@LaFayette great progress! Can we take this opportunity to make the version be a string so that it can match the version in the map itself instead of being an integer? especially since the UI renders it as a semantic version when it's not. As far as if the map needs to be updated, a simple inequality check should work great and allow map makers to version the map however they see fit.
-
7 items in done column, 3 in WIP (work in progress), 4 still be started with potentially more to be added in the 'to-be-started' (TODO) column. Still quite a bit to do. Getting it so the game-engine used 'map.yml' file is crossing the line though, it's a major milestone.
Please focus on the 'map.yml' file structure, names, specifics and anything that could be missing or needs to be otherwise specified. This is essentially the LAST chance to give feedback on the map.yml file (not when 2.6 is released, but now). Once we have created one and uploaded it to the 100+ map repositories, it'll be painful to change and probably won't unless it must.
-
@djabwana The version value in the map (game) XML is deprecated. Even in 2.5 and before it's not used for anything (it is display ony). In 2.6, the GV column has already been removed from the lobby table and the XML attribute for it will be ignored.
The version value in the download description is another entity and exists in 2.5 (in triplea_maps.yaml). It is only used to notify players that they should update.
One issue with the XML version is that there are multiple XMLs to a map, yet the map needs a single version to be able to notify players of a new updated map download.
Can you suggest a further utility for a semver? Can we solve the problem of also explaining semver in a consistent manner and also having it be used in a consistent way (in other words, if we use a semver, how are we sure that everyone understands it and uses it in a consistent way)? IF we go to semver, how would the engine make further use of it?
The version in the map download description is used by the engine, so it is important that it is used in a consistent and predictable way.
-
@LaFayette Gotcha. My point is that map makers should be able to sem ver if they want to instead of being forced into an integer, so that way if they're like me and they want to tag the source repo and use Maven and stuff (even apart from it being removed from the XML) there won't be a mismatch with the "real" version.
People can use sem ver, but if the version is any old string they can do whatever they want. By opening it up, the version could be
1.0-new-unitsand that would show up in the map list dialog, the lobby, etc. since it would be the real version not just a discarded XML thing.I think the integer is overly restrictive, but the upside I'm guessing is comparability? so you can say > or < on the versions? But I was proposing we just open it up and do a simple !=, and it's up to the map maker to ensure that makes sense (e.g. not regress back to 1.0 after they released 2.0 - but they can if it turns out 2.0 was no good...)
-
There are a number of considerations here. First, should the download version and the version of the map be different version values? If so, what is the utility?
Second, map makers should strive to strenously avoid any incompatible changes. A series of problems arise as the engine is just not designed for players to have anything other than the latest of a map. This aspect defeats most of the benefit of a classic semver.
Next, the utility vs complexity trade-off, what is the benefit of a semver compared to the complexities? The complexities include:
- having to explain semver
- having to know what an incompatible change is (which is nontrivial, should be understood anyways, and incompatible changes should be avoided!)
- being able to learn by example, we could have a "wild-west" of version values otherwise
Another aspect is reconciling the many to one relationship between XML versions and map. The map version really needs to be its own thing so the engine knows whether a player has an out-dated version. Having the map version track against multiple XML versions gets complex.
Yet another aspect is version value management. It is bad practice to have version be maintained in multiple places. If the engine were to derive the latest version from say git tags, that would be quite interesting and would avoid having to record the version value anywhere at all. On the other hand, that introduces its own complexities and we are strenuously optimizing for simplicity when it comes to map upload (there is a lot of resistance to git repositories, starting with having to create an account). On the other hand we can flip the perspective and use feature branches to control non-latest versions and then the engine only needs to care about the latest merged into master. The engine could perhaps automatically pick up the last 'sha' of master and record that as a version. Then a downside there is players are notified whenever there is any change to map (which would be okay if using feature branches were a more common thing, but that introduces more learning curve and is therefore avoided)
-
I think we're missing each other here. I'm not at all arguing for sem ver, and I 100% am arguing for not having two different versions. All I'm arguing for is switching it from an
intto aStringin the YAML. -
Sorry, I may be confused. Wouldn't a
Stringbe there in order to have a semver? Unless I am totally confused, I realize you are suggesting it be optional for map makers to use semver. Though, I think that creates some complexities that would need to be addressed.
)){kind=link}