Storing Map Index in Database - Deprecating triplea_maps.yaml


  • Admin

    I'm working on a few things that I hope map makers will really like. I'm opening this thread to discuss the details while still in design and building out the code.

    The initial rev will focus on automating the reading and writing of: https://github.com/triplea-game/triplea/blob/master/triplea_maps.yaml

    Currently forecasting this to be a 2.2 or a 2.3 feature.

    Naming the maps server

    I'm gonna start with the most important : )

    The lobby server is currently code-named 'spitfire'. We're going to have a second server for maps, I'm thinking to call it the 'stuka' server.

    Github Project

    FWIW, this is the github project link where progress can be observed: https://github.com/triplea-game/triplea/projects/17

    All dev issues regarding that project can be found with this link: https://github.com/triplea-game/triplea/issues?q=is%3Aopen+is%3Aissue+project%3Atriplea-game%2Ftriplea%2F17

    Getting map list from Stuka

    The first thing is the game engine will be updated to pull its map list from stuka instead of reading a github file (https://github.com/triplea-game/triplea/blob/master/triplea_maps.yaml). The stuka server would a database to generate the list of maps and return that back to the game client. Essentially this will be exactly the same info as found in the YAML file, but retrieved from database

    Pushing update to Stuka Map Index

    This is where it gets good. The next step will be to add a UI to triplea-game where a person can open a menu and fill in fields needed for a map version. First you'll enter credentials same you would to log into the lobby server. Then you'll see a UI where you'll be given a list of existing maps that you can update or you can upload a new map. You'll then be given another UI to either publish an update to an existing map or fill in fields for a new map. You can then click 'publish' and your map index will be created in database (no more PRs to update the YAML file).

    From then on, anyone downloading maps will see the new or updated map.

    For some period of time the server will continue to read the YAML file on github, so we'll still have a legacy route where if you update the YAML file the server will be (eventually) updated. This file will disappear once all clients are migrated to using Stuka.

    Considerations

    Map Ownership

    Map updates will be limited to only those that uploaded them. We'll need to assign owners to each of the maps. This will probably be seeded by the current owners, though most are just the map admins. Anyone who is an 'owner' can update a map.

    Sharing Maps

    I'm thinking it will be nice to be able to have multiple people be able to push to a map. I'm thinking we can add a UI option for a map owner to add additional people who are allowed to update their map (remember for now it's just the map index information, so not as profound yet as actually just updating the XML, but we'll be getting there 😁 )

    Map QA & Moderation

    An important element here is that new maps can just start appearing on the system. This is potentially bad from a moderation perspective (hopefully nobody will be quite this bored, but bad things can happen). Currently we are gatekeeping maps as we control when a map is updated, when the download URL is updated, and we can see whenever any map contents are changed. This will be relaxed and we won't be able to control when maps are added.

    To this extent, we'll probably need a 'moderation' flag on maps, to indicate that maps a 'raw' and could contain anything. I'm not sure if we want to by-default make these available to the public or require that a map-moderator look at the map first and flag it's okay. The latter is a bottleneck and requires resources. We can perhaps label a pretty large group of users as 'trusted' and allow them to do the map-moderation.

    Map Versioning

    Currently the engine only supports one version of any map, the latest. I'm intending for this to be relaxed and for the engine to be able to manage multiple versions of a map. The different versions will continue to be transparent to the game engine and it'll work only with the latest, but we're laying the foundations early to support multiple map versions.


  • Admin

    Some thoughts from my side:
    I like the code-names, they add to the feeling of a modern system.

    First you'll enter credentials same you would to log into the lobby server.

    At this point I'm wondering where servers are separated.
    I'm wondering if it would make sense to create a standalone auth server with the single purpose, which is then used by all services that need auth.

    Regarding the moderation:
    I think having a similar system to the one wikipedia uses for editing articles might be a good move:
    Basically anyone can edit or publish a map (editing of course only by "owners" in our case) and the changes are immediately visible on a "not-moderated" changes page, which is opt-in. However until a moderator approves the map, the old listing will be the one that is first displayed to users. WDYT?


  • Admin

    Going down the micro-service route could likely hurt us more than it helps. I wouldn't want to break out functionality to that low of a level without additional reason to do so.

    @RoiEX the moderation aspect you're suggesting is pretty close to what I'm thinking. The trusted users would have a queue of maps and they'd approve them at which point they would become live. Though, I am nervous about the overhead and complexity that could create. If we lock down the map URLs to github based ones, then we can delay that feature for a little while at least.


  • Moderators Admin

    @LaFayette I'm never sure, due to the traditional terminology being not always followed: By "Map Versioning" do you mean that the maps will have a version or are you referring to the version numbers within the game (xml) files? I'm guessing you are referring to the actual map, and I, then, suppose that the map folder name will change with every version, so to allow to have more than one of the same map on your terminal (similarly to when, in the past, new TripleA instalments were not overwriting any previous ones)?

    A thing I always disliked about the current map updating system (beside the time inconsistency due to the time between the map update and the moment in which someone merges your PR to update the "triplea_maps.yaml") is the update versioning number, that it is itself meaningless but for prompting users to update. Will the process be the same (meaning the first time you add a map it will be version 1, the first update will be version 2, and so on)? In the moment that was not really a map versioning number, I'd rather keep track of it by registering the date in which the update was made (maybe limiting to no more than 1 update per day), which would also be a valuable information per se.

    If there is a map versioning akin to the release versioning in the current "triplea_maps.yaml" (meaning the first time you add a map it will be version 1, the first update will be version 2, and so on), I think it would be better the version being kept within the map folder (so it will be an actual map versioning) and a versioning increase being strictly required every time anything part of the map is changed (so that, if you change any game file, you have to increase both the game version and the map version, while, if you change only any element outside all game files, you have to increase the map version but not the version of any game).

    Just some thoughts, not sure how much on topic, as I'm not really understanding everything that is being said about this new process.


  • Admin

    responding to your various questions/concerns @Cernel

    due to the traditional terminology being not always followed.

    The traditional terminology is frankly wrong and made up, hence the confusion - but that is really besides the point.

    By "Map Versioning" do you mean that the maps will have a version or are you referring to the version numbers within the game (xml) files?

    The version number in the XML files would go away completely and the entire map would be versioned.

    the map folder name will change with every version, so to allow to have more than one of the same map on your terminal (similarly to when, in the past, new TripleA instalments were not overwriting any previous ones)?

    TBD, but likely so.

    A thing I always disliked about the current map updating system (beside the time inconsistency due to the time between the map update and the moment in which someone merges your PR to update the "triplea_maps.yaml") is the update versioning number, that it is itself meaningless but for prompting users to update.

    Yup, the version number in the triplea_maps.yaml is purely for notification purposes. It's much more involved to know the last version that users had and the existing game code has no concept of map versions. (Game versions are also there just for display purposes, that does not even trigger anything, it's only used for display in the lobby 'GV' column).

    Will the process be the same (meaning the first time you add a map it will be version 1, the first update will be version 2, and so on)? In the moment that was not really a map versioning number, I'd rather keep track of it by registering the date in which the update was made (maybe limiting to no more than 1 update per day), which would also be a valuable information per se.

    When a map is updated would be tracked. A single incrementing sequence for versions is likely. We are getting a bit ahead of ourselves as the game engine does not support multiple map versions. How that is to be done is TBD.

    We could easily get into a state where a map truly becomes the XML file and the assets are decoupled from the map. Notably we could have a 'unit editor' feature where unit images, flags, resources, etc are all uploaded individually. Then a game would reference such elements by ID value. When you launch a game you'll have a units folder that has a collection of unit IDs and they are brought in as needed. This avoids all overlap between map resources and allows for map-makers to pick and choose which graphics they would want. Same thing could be done for map polygons. For now though the focus is on deprecating triplea_maps.yaml with a for-like replacement that can be interacted with from in-game.

    If there is a map versioning akin to the release versioning in the current "triplea_maps.yaml" (meaning the first time you add a map it will be version 1, the first update will be version 2, and so on), I think it would be better the version being kept within the map folder

    I don't think that is readily possible. What if two people update a map at the same time? How do you know a version before you go to upload it? What happens if one version is worked on for a long time and you then do not know the right version number. The version number I think must be external to the map, it should probably not be aware of any other map variants and just worry about being complete in of itself. Any version matching would then be the responsibility of tracking and the game engine.

    (so it will be an actual map versioning) and a versioning increase being strictly required every time anything part of the map is changed

    Very likely each map upload will be write-once and then read-only (immutable). When uploaded it would get a tracking ID and a version number to know which ones are which and which is latest.

    you have to increase both the game version and the map version

    That almost certainly will be unnecessary and both versions are just going to be removed from the map-maker domain.

    if you change only any element outside all game files, you have to increase the map version but not the version of any game).

    A map and its games are going to be more tightly bound. Perhaps even moving to where they are 1:1 and then it becomes a needless distinction.

    as I'm not really understanding everything that is being said about this new process.

    What is not clear?


Log in to reply