Proposal to consolidate Unit options into UnitList & Make explicit which units are available to which players

LaFayette

This proposal is spawned from a game performance problem that in part stemmed from there being no concrete association between unit types and players.

https://github.com/triplea-game/triplea/issues/7142

@Cernel said there:

For example, this way, in World At War, you would have the "red dot" units in unit help only if you open it after the start of round 4. If you want to know ahead what are the abilities of the units that you will have, it makes sense that the notes should do this (and the notes of World At War do it fully and flawlessly, except only for the fact that they don't display all unit images, but you need this only once the units are actually on the board (at which point you have them in Unit Help)).

Long story short, out of this:
    all units on map
    all production frontier
    all unit artworks
I suggest completely to eliminate the third item.

Unit help is meant to be all units, future and present. This way you can plan which units you want to purchase before they are available.
Game notes is responsible for doing a lot already, avoiding duplication between game notes and the game functionality is a good thing. We want map making to be easy & light weight.
The unit help is tied to the unit stats exports, they're meant to be roughly the same

If we have an explicit tie in XML, then all three mechanisms, units on map, production frontiers, and unit artwork would go away. We'd also be able to determine when a map launches exactly if it has all of the right images. Another relatively major problem is the engine does not know which map images are needed until they are needed. If the engine could determine that at game-launch, it'd remove a class of error and would further improve map making.

Pending further ideas, I'm strongly in favor of XML being explicit about unit-type relationship, it seems to solve a number of problems that I otherwise do not really see how else to cleanly solve.

Cernel

@LaFayette said in Proposal to consolidate Unit options into UnitList & Make explicit which units are available to which players:

Pending further ideas, I'm strongly in favor of XML being explicit about unit-type relationship, it seems to solve a number of problems that I otherwise do not really see how else to cleanly solve.

Would this be merely an "information" feature, irrelevant for gameplay? Am I correct to assume that it would matter only for new or specifically updated games?

alkexr

@LaFayette I have minor disagreements about this syntax. First, the list of availableTo players should be a colon delimited list, as those can generally be edited much quicker than copy-pasting (single-line) xml blocks. But my biggest concern is that moving unit attachments out of the attachmentList means that variables and foreach patterns can no longer be used for units. Now, unit attachments aren't the primary candidates for these, but I would say it's still a bad direction, unless variables are extended to the unitList as well. There is also a case to be made that units belong to players, not the other way around, so instead of availableTo, players should have a list of availableUnits. As a mapmaker handling maps with 50+ different units I would much rather edit one player's available units in one place if possible.

LaFayette

I appreciate the discussion @alkexr , the point of this discussion is feedback and to decide the direction.

First, the list of availableTo players should be a colon delimited list

While that can be the case, I would respectfully point out that is not canonical XML. FWIW, XML models lists as:

<elements>
   <element>value1</element>
   <element>value2</element>
</elements>

You can always convert such a list to a colon delimited list provided that colon is an invalid character in any list entry. One thing you cannot do is then augment your list to have more attributes or values, for example:

<elements>
   <element>
      <name>key1</name>
      <value>value1</value>
   </element>
   <element>
      <name>key2</name>
      <value>value2</value>
   </element>
</elements>

As convenient as it might be, a colon delimited list is arguably bad XML practice as XML can and does model lists. List values are not meant to be special encoded attribute values, attribute values are quite meant to be scalar values.

The above you can copy/paste easily to additional lines, it's debatable if that's easier than finding the right spot to add a new entry.

I suspect this is a pedantic point, but it does kinda speak to the overall XML modelling being very non-canonical. There is a cost there as the less than ideal model makes parsing difficult and constrution of an XML non-trivial as well as there is a lot of cross-reference and duplication.

But my biggest concern is that moving unit attachments out of the attachmentList means that variables and foreach patterns can no longer be used for units

What would imply that? A canonical way to do the variable list would be to make it a first class element and then reference it with a specific tag, eg:

      <playerList name="allies">
              <player>Americans</player>
              <player>British</player>
              <player>Russians</player>
       </playerList>  

      <availableTo>
              <playerList>allies</player>
       </availableTo>

In all practicality we might want to re-use the variable mechanism, though the above is more first class way to support a variable list. From a technical perspective we've been seeing issues parsing XML maps, the variable expansion might not necessary be optimally done, though it could debatably be simpler to re-use that as-is. One downside is you do not get as much of a syntax check. In the above you can get a clear error message if you typo the "allies" player list name for something that does not exist, where-as with variable expansion you won't get the same type guarantees. That does make XML editting more error-prone and harder to debug as the error messages are simply not there.

There is also a case to be made that units belong to players, not the other way around, so instead of availableTo, players should have a list of availableUnits.

I think this is perhaps the primary modeling consideration. It looks like we are going to be forced to duplicate either the unit type names, or the player names. Perhaps we'll need to consider how much more future consolidation and overall direction we would want to go.

I'm only nervous about that approach as a grand-update would make for a much more difficult lift, a steeper learning curve, and make it just less likely we'll complete the effort. Meanwhile it is a problem that the game engine is not told anywhere which units belong to which players (which leaves utterly hacky and incorrect code for the engine to figure that out; it needs to be made explicit).

As a mapmaker handling maps with 50+ different units I would much rather edit one player's available units in one place if possible.

This is an interesting scenario when the unit list is large. I'm not sure if it actually helps that much when you have 50 units listed under each player. This implies as well more of a white list. Would it not help that by default all units are available to all players? The time to specify when a unit is available to specific players is only when that unit is not available to all players. If you add a unit, that is available to all players, it would be just one line. IF there are 20 players, and we list all units, say 30 units, then you've got 600 lines to maintainer. Add a new unit, you add 20 lines, add a new player, you add 30 lines. ON the other hand, if we do units to players, you add a new unit, it's one new line. If you add a new unit that is available to 19 out of 20 players, then you add 1 line for the unit and the 19 player entries (and here we can make use of lists if very likely the unit should only be available to an alliance). IF the unit is player specific, then we're only adding one or two player entries, which keeps it pretty manageable.

Cernel

@LaFayette Variables are terribly inefficient, currently. A game file (xml) with variables parses much slower than the same file without variables and the same functionalities. However, I've always assumed @redrum is aware of this and it doesn't actually matter as long as the file is not too big (probably creating the code from the variables has to be time consuming, I guess).

LaFayette

Three quick observations:

general variable usage disallows type checking. For example, you can use a variable for resource types and use that as a player list. If we were to have a 'playerList' tag with a name, the engine could know that it would be an error to have that list be used for a list of resources.
We currently can't use variable lists for production frontiers
Looking at WorldAtWar xml, I'm seeing neutral production frontiers that are not attached to any player, it looks like we've fallen into a not great situation with unlinked entities.

LaFayette

@Cernel I'm not sure why it would be inefficient. One problem is that variables imply a complicated map, which then implies a long XML. The XMLs are currently longer than they need to be due to the modelling, just not as expressive as it could be (IE: we should not need a 30k line long XML, I understand some maps are complicated, though that is an issue with the XML I think and not the complexity).

Cernel

@LaFayette Well, I only know that if you make a game file with some functionalities and, then, reduce the same file, keeping the same functionalities with less code, by the use of variables, the file using variables is a lot slower to load and parse than the one without variables. I mean "a lot".

LaFayette

@Cernel That's unfortunate. Perhaps there is something that can be readily fixed in there. Otherwise modelling lists "more properly" would resolve that. Essentially instead of a variable one would create an XML element and then reference that element (EG: the playerList element above, instead of a <var name='allies' values='russia,uk'>, you create the <playerList name='allies'> tag instead.

wc_sumpton

@LaFayette said in Proposal to consolidate Unit options into UnitList & Make explicit which units are available to which players:

This implies as well more of a white list.

Yes it does, and a necessary evil with the xml. Take TWW as an example, for Germany you will find germanInfantry, germanTank, german... etc. The same for the Russians, russianInfantry, russianTank, russian... etc. Then looking at the unitAttachment's for germanInfantry, russianInfantry they are the same. It is only during game play when changes are made to one player's unit but not to all players.

Lets say that Germany's Tank will receive an increase in offensive firepower. With techAbilityAttachment that can be done. But techAbilityAttachment is very limited in what can be changed.

If I wanted to give Germany's Infantry the ability to amphibious assault from a German Carrier. Well it can't be done unless you remove both carrier and infantry from the 'white list' and modify germanInfantry and germanCarrier.

So X+ similar unit per player has become the norm.

Cheers...

alkexr

@LaFayette What I meant is that to my knowledge variables can only be used within the attachmentList right now. Also, colon delimited lists are a lot more convenient right now (they accept variables). So I'm approaching this from a baby steps perspective. And yes, I'm prefectly well aware what the ideal xml structure is: it's a remarkably horrible way of representing the newer generations of TripleA maps, even if it used to be just fine for A&A et al. Even the concept of variableList is more or less just an outside hack that I originally implemented as a standalone script to generate the xml file, because I couldn't bear its clumsiness. The ultimate solution would of course be a script language for TripleA, but yeah, I'm not going to create that either (or not at the moment anyway).

LaFayette

@wc_sumpton I think you might be pointing to a problem regardless of how it's done with this proposal. It might be cleaner with the explicit list, but it's probably equivalent.

Either:

<playerList>
   <player name="German">
        <units>
            <unit>GermanTank</unit>
        </units>
   </player>
<playerList>

Or:

<unitList>
   <unit name="GermanTank">
       <allowedPlayers>
             <player>German</player>
       </allowedPlayers>
 </unitList>

In either case IMO the special attribute or unit behavior is not allowed to be player specific, so create a specific unit for it seems to be the core problem. For either modeling variant it's kinda equal.

One difference in the above is how the production rule would be nested (or not nested). That would make the latter more attractive.

LaFayette

@alkexr I'm wondering if actually maybe a grand re-write is actually the way to go. Supporting many small XML spec variants would not be that easy.

RE: colon lists

I'm on the fence here. On the one side it is convenient, on the other it's debatable and there is so much less than ideal structuring that it would be nice to draw a line and start doing canonical XML. Canonical XML is quite verbose, but it "keeps you out of trouble" and keeps parsing more sane and generally more performant. The encoding of attributes via colons makes for not great code and there are numerous places where you don't get any error message at all for invalid configuration. A more canonical structure would get us there.

LaFayette

Here is an example of latest proposal:

Example of current:

  <unitList>
    <unit name="Stuka"/>
  </unitList>

  <production>
    <productionRule name="buyStuka">
      <cost resource="PUs" quantity="8"/>
      <result resourceOrUnit="Stuka" quantity="1"/>
    </productionRule>
  <production>

  <productionFrontier name="GermansFrontier">
    <frontierRules name="buyInfantry"/>
  </productionFrontier>

  <playerProduction player="Germans" frontier="GermansFrontier"/>


  <attachmentList>
    <attachment name="unitAttachment" attachTo="Stuka" javaClass="games.strategy.triplea.attachments.UnitAttachment" type="unitType">
      <option name="movement" value="4"/>
      <option name="attack" value="3"/>
      <option name="defense" value="1"/>
      <option name="artillery" value="true"/>
      <option name="isAir" value="true"/>
    </attachment>
  </attachmentList>

Example of proposed:

  <unitList>
    <unit name="Stuka">
        <allowedPlayers*>
           <player>Germany</player>
           <playerList>Axis<playerList>
        </allowedPlayers>

        <productionRules>
            <productionRule>
               <costs>
                  <cost resource="PUs">8</cost>
               </costs>
               <players*>
                 <player>Germany</player>
                 <playerList>Axis<playerList>
               </players>
            </productionRule>
        </productionRules>

        <stats>
          <movement>4</movement>
          <attack>4</attack>
          <defense>1</attack>
        </stats>

        <options*>
           <option name="artillery"/>
           <option name="isAir"/>
        </options>

    </unit>
  </unitList>

(* = optional attributes)

Comments/notes:

using player list we can consolidate all production rules to just the above, the current example would more properly bring in every player that has the production frontier
the above has a lot less cross-references, no need to name frontiers and then link them. No need to cross-reference the name of a unit.
we do have cross-references to players and the playerlist. I suspect the above would scale pretty okay if you add either a player or a unit (notably, adding a unit, it's just a single block of XML that is added).

If we go with the grand re-write, YAML is an option. In YAML the equivalent configuration would be:

unitList:
  - name: stuka
    allowedPlayers*:
       players: [Germany]
       playerLists: [axis]
    productionRules:
      - name: stukaPurchase
        costs: 
           - {resource: PUs, quantity 8}
        players*: [Germany]
        playerLists*: [axis]
    stats: {movement: 4, attack: 4, defense: 1}
    options*: [ isArtillery, isAir ]

Cernel

@LaFayette You should also be able to purchase resources or a mix of units and resources.

For example, with 1 PUs you purchase a "stuka" unit and 1 "a" resource. You can also purchase 2 PUs with 1 PUs, that would equal an investment with a 100% interest rate over 1 round.

alkexr

@LaFayette The YAML is about ten times more readable. Stats or options being in a single line isn't that great though. Think some unit with AA attacks and AA attack max dice sides and bombing damage and all that, I'd definitely want to group AA related stats/options or bombing related ones or construction related ones together.

Also consider that some maps use multiple production frontiers and swap them during game using triggers.

LaFayette

@Cernel Costs being a list allowed for multiple cost elements, eg:

      costs: 
        - {resource: PUs, quantity: 8}
        - {resource: Oil, quantity: 2}

For purchases resources, good consideration if we are going to remove the production frontier tag. To replace it we'd want to update the resource list and add a similar production rule to that.

@All/

At this point, despite my earlier reservation, I think there are two larger questions at play now:

1. Do we do a grand update of the map spec? Is a much larger project in order?
1. Do we consider using YAML as the updated spec? Splitting the game engine parsing logic to keep the existing XMLs the same and to have a new parsing logic module for YAML would arguably be easier than having multiple XML versions.

LaFayette

@alkexr good considerations. We can certainly add/create attributes for AA, eg:

   attributes: [ artillery, air ]
   aa_attacks: { count: 3, dice_sides: 6 }
   bombing_damage: {dice_sides: 10, number_rolls: 3}

It's a bit limited to make all attributes just be an 'option' tag. The XML doctype makes that kind of move attractive. Moving to YAML makes that a lot more fluid, we could model options much more accurately.

For frontiers being turned on and off, we could add a triggers attribute to the production rule, and/or add a name field to it so it can still be referenced from triggers.

Cernel

@LaFayette said in Proposal to consolidate Unit options into UnitList & Make explicit which units are available to which players:

Splitting the game engine parsing logic to keep the existing XMLs the same and to have a new parsing logic module for YAML would arguably be easier than having multiple XML versions.

This seems the best solution to me. Hopefully the YAML would be faster to parse and load (also for the battlecalculator)?

As long as all that it is possible to do now with the XML will still be possible in the new format, it should be fine for me.

Cernel

@LaFayette As I said, I would advice any such things working like the support attachments currently do (this way you could also have multiple different AA abilities per unit). Meaning you have the ability that you define and give it to a list of units and a list of players.