Should we support YAML for Maps?

[LaFayette]

YAML can be thought of as a lighter weight and more human friendly variant of XML.

For example:

<players>
   <player>British</player>
   <player>German</player>
</players>

players:
  - British
  - German

We have a few options going forward. Fundamentally the big question is if we should add support for defining maps in YAML. In the works is something that will make map making better. The XML can be made more expressiveness and some large improvements are due. For options, we ahve:

A, stick with XML

We improve the current XML and allow for a more expressive XML variant. IE: we improve the existing XML so you can write a version of it that is more compact and more expressive

B, go to YAML sooner

Do the heavy lift of the simplifications and grand rewrite in YAML directly essentially leaving the current XML behind as frozen.

C, do grand rewrite now, support YAML eventually (both A & B )

This path has us improving the current XML quite a bit but we keep YAML in the radar and seek to create a parser that will support YAML.

Pros / Cons

(A) is seemingly the default path that we will be taking.

(B) has the drawback that it could be a while and meanwhile the current XML languishes. There is also a learning cost for people to learn YAML. We'll also likely never get rid of XML so we'll always have both XML and YAML.

(C) We can certainly plan to allow for YAML eventually, the drawback here is difficult support.

If we build a YAML processor, likely new features would only land in YAML. We'll also need to consider support cost, it could be more difficult with more variations to keep the code simpler and to help people when they have map making problems.

What are people's thoughts? Is the simpler and more expressive YAML worth the transition cost? If we do a big simplification of the current XML, should we go directly to YAML or do the simplification alternative in the current XML so you can either user the "old" XML or "new?"

[ubernaut]

@LaFayette i would say if it's easier for new people to learn, it would probably be worth it but i have no idea what the codebase looks like or anything about programming in general anyway so yeah fwiw…

[ff03k64]

Part of the question would be how long would the options take. I am guessing that switching to YAML would basically stall most other progress until it can do what XML can do now? But maybe that would happen in improving the XML as well.

Just how much better is YAML than XML? How much of the improvement is human friendly, and how much is YAML actually being able to do more? Human friendly is nice, but in my map, a lot of what i am doing so far is copying and pasting, and then just changing the variables. So your small example wouldn't really be a big change. It would get rid of some common errors (stupid missing /s), but I am guessing it would have its own set of common errors.

So far as new features only landing in YAML, once YAML is done, i would guess that new map makers would learn in YAML instead of XML. So how hard is it to transition between from XML to YAML?

If you switch to YAML, would that involve any engine changes? I am guessing there are plenty of things that could be done cleaner, maybe switching to YAML would be a good time to do that? But if it is basically just making a new parser, that doesn't really change things on that end.

[LaFayette]

Good questions @ff03k64

The answers are not short though, there is a lot going on. Updating maps and how they are parsed fits into other projects like a pretty big puzzle, eventually save games will be more compatible and maps uploaded from server. This all feeds into that. The question of whether to YAML or not, is not an easy one.

To start answering, a lot of benefit we get will just be purely from a more expressive structure. XML and YAML are largely equivalent, you can represent just about any XML in YAML and vice versa.

YAML is just less verbose, has no end tags, does not need quotes all over the place, and can be a lot more compact. In particular YAML is good for nested data. In YAML there are three main data types, scalar (a single value), lists, and maps. Maps is a list of key, value pairs, a list is just a listing of values, and a scalar is just a single value. In YAML you can nest these quite easily to have a list of maps which then have as their values more lists.

Let's look at an example, in XML today we have effectively the following map data structure in a 'step' (it's a name and then a series of key value pairs):

<step 
   name="step1" 
   delegate="stepDelegate1" 
   player="player1" 
   maxRunCount="1" 
   display="stepDisplay"/>

In YAML:

step:
  name: step1
  delegate: stepDelegate1
  player: player1
  maxRunCount: 1
  display: stepDisplay 

A list in XML is really particularly verbose, eg:

        <map>
                <!-- Territory Definitions -->
                <territory name="East Prussia"/>
                <territory name="Pomerania"/>
                <territory name="Saxony"/>
                <territory name="Berlin"/>
                <territory name="Rhine"/>

In YAML (probably would make sense to rename 'map' to 'territories' here):

map:
  - East Prussia
  - Pomerania
  - Saxony
  - Berlin
  - Rhine

An example of how we should make the XML more expressive would be the connections. For example today we have something like:

        <map>
                <!-- Territory Definitions -->
                <territory name="East Prussia"/>
         </map>
         <connections>
                <connection t1="East Prussia" t2="Lithuania"/>
                <connection t1="East Prussia" t2="24 Sea Zone"/>
                <connection t1="East Prussia" t2="Brest"/>
                <connection t1="East Prussia" t2="Danzig"/>
                <connection t1="East Prussia" t2="Latvia"/>

As more expressive XML, it would be:

     <territories>
                <territory name="East Prussia">
                      <connection to="Lithuania"/>
                      <connection to="24 Sea Zone"/>
                      <connection to="Brest"/>
                      <connection to="Danzig"/>
                      <connection to="Latvia"/>
                </territory>

As YAML:

territories:
  - name: East Prussia
    connections:
     - Lithuania
     - 24 Sea Zone
     - Brest
     - Danzig
     - Latvia

There is a short-hand notation in YAML as well where you can define the data in one line, eg:

territories:
  - { name: East Prussia, connections: [ "Lithuania",  "24 Sea Zone", "Brest" ] }

The most common pitfall in YAML is the indentation, it is significant and items don't line up you'll get an error. For example:

territories:
  - name: East Prussia
    connections:
  - Lithuania
     - 24 Sea Zone
     - Brest
     - Danzig
     - Latvia

In the above, lithuania is badly indented, YAML will think that it is another item in the 'territories' list rather than as an item in the connections list. Then 24 Sea Zone will generate an error as it's an illegal start of a list. The prevoius element 'Lithuania' is a scalar value, you can't have a list following a scalar value.

So far as new features only landing in YAML, once YAML is done, i would guess that new map makers would learn in YAML instead of XML. So how hard is it to transition between from XML to YAML?

Given the skill of people having learned XML so well, YAML is not tough. For new map-makers, hard to say, would maybe be slightly more difficult because it is less structured but perhaps about the same.

If you switch to YAML, would that involve any engine changes?

Oh yeah! We'd have to have two completely different parsers. I'm working on some updates that make the engine parsing to be two step. Today it is one step where in one swoop we parse the XML and create game data entities out of it. The updates would make it two step and introduce an intermediary layer that models the data that is in the XML file. The first step in parsing will then be to convert the XML into this intermediary layer. The second step will be to convert the intermediary layer into game entity objects ("game data assembly"). This update will go a long way to either direction as the "game data assembly" will always be the same, the parser of whatever we do will just need to create the same intermediary data and from there it'll be existing code that knows how to assemble it.

With all that said, the difference is going to be whether we have a lot of XML parsing code that says "go here to get this data OR go here to get it", vs having a completely different parsing mechanism. With the intermediary layer either option should not be too bad and hence why some more significant updates are being discussed/proposed. It'll soon be much more feasible to read for example territory connection from either nested 'connection' tags or to read them from their current location.

I am guessing there are plenty of things that could be done cleaner, maybe switching to YAML would be a good time to do that?

The way XML is being read is being cleaned up anyways. The intermediary layer described above fits into other projects to help make game saves easier and more compatible and also make it easier/possible to upload maps from the game itself!

For the XML itself, it's a good time to make it more expressive. Whether we do that in the XML, as YAML, or both is really an open question.

[Cernel]

@LaFayette said in Should we support YAML for Maps?:

probably would make sense to rename 'map' to 'territories' here

It would be better first clarifying exactly what a territory is. As far as I know, a territory is a zone that can be owned (actually, TripleA wise, that is a zone that is always owned, as it is owned by the "Neutral" (null) pseudo-player, when it should be unowned). If so, those should be called "zones", not "territories" (the "zones" would be all the named areas on the map, while the "territories" would be only the zones each of which has a territory attachment). However, TripleA is quite consistent in, apparently, using the term territory just to mean zone, so the matter, as I said, would be giving an official TripleA definition of territory, first, unless I'm overlooking something (also clarifying whether or not sea zones can be territories too (if not, then also the "territory" attachments should not be called "territory", since they can be assigned to sea zones)).

[LaFayette]

@Cernel discussions on renaming elements will come later. Having a list of territory names under a "map:" list is kinda confusing, but it is just completely besides the point.

The decision to go to YAML is very important. It is something we will live with for years to come.

[beelee]

well, it looks as if it it would be easier to do stuff with yaml. You'd need a POS2 done in yaml so people would have a starting point.

However, it seems as if knowing how to use xml may be a better overall skill to have. It seems closer to actual java language type stuff. I could be wrong, as I know almost nothing about java. I did find a program that changes xml into java and vice versa but haven't tried it.

I struggled learning the xml and there still is a lot I don't understand. It seems there are quite a few helpful people on the site when it comes to working with the xml. Map making requires dedication, so if there is a bigger learning bonus by working with xml, I'd like to see it kept for a learning type tool.

If any of that makes any sense

[Cernel]

@LaFayette said in Should we support YAML for Maps?:

The decision to go to YAML is very important. It is something we will live with for years to come.

The only major things I care is that everything currently possible will be still possible and loading the game from the game file (YAML) will not be slower (I'm thinking about loading huge files, like "War of the Relics").

[ff03k64]

I have been reading and thinking, and still wish I understood the difference between the two better.

I think being forced into certain indentation to function probably will cause almost as many issues as I am running into with xml right now.

I think the best thing that could be done is to get good error logging for trouble shooting. Some are great where it says what line is causing the error, but others need a lot more knowledge than I have.

Do we have a ranking of YAML's advantages? Ease of use for map makers was one of those advantages. A good tutorial map could go a long ways towards that as well, probably with less work?

[LaFayette]

Good questions/considerations again @ff03k64

I kinda like this quote:
"XML is useless garbage. It's dated, so don't use it."
https://www.quora.com/What-situation-would-you-use-YAML-instead-of-JSON-or-XML

In essence, XML was an early standardized format that grew out of HTML that was meant for structuring data for one computer to send data to another. It was never really intended for humans to interact with it and write it by hand (the idea was there would be GUIs to help you do that).

Currently, JSON is preferred now when sending data between computers. JSON is also painful to work with by hand and read, and that is where YAML steps in. That no longer leaves any room for XML.

From: https://www.csestack.org/yaml-vs-json-vs-xml-difference/

• YAML stands for “YAML Aint Markup Language“.
• YAML uses indentation to define structured data. So each block in the YAML is differentiated by the number of white spaces.

• XML is “eXtensible Markup Language” whereas YML is not a markup language.-
• XML uses a tag to define the structure just like HTML.

In short though, YAML is just more expressive, you don't have the end tag, you don't need as much quoting, and you can nest data structures in a more concise manner. You also don't need as many intermediate tags in YAML for lists, for example:

  <xmlList>
        <element>1</element>
        <element>2</element>
  </xmlList>

vs:

yamlList:
   - 1
   - 2

Or in YAML short-hand:

 yamlListShortHand: [ 1, 2 ]

As a more involved example, YAML:

units:
  - name: Infantry
    cost: {resource: PUs, quantity: 2}
    image: infantry.png
    initialPlacements:
      - {territory: "Western USA", quantity: 2}
      - {territory: "Western USA", quantity: 2, owner="British"}
      - {territory: "Eastern USA",  quantity: 10}
  - name: Stuka
     cost: [ {resource: PUs, quantity: 8} , {resource: Oil, quantity: 1}}
     availableTo: Germany
     image: stuka.png
     initialPlacements:
        - {territory: "Southern Europe", quantity: 2}
        - {territory: "Germany",  quantity: 10}

As XML:

<units>

   <unit>
      <name>Infantry</name>
      <costs>
         <cost>
             <resource>PU</resource>
             <quantity>2</quantity>
         </cost>
       </costs>
      <image>infantry.png</image>
      <initialPlacements>
        <initialPlacement>
           <territory>Western USA</territory>
           <quantity>2</quantity>
        </initialPlacement>
        <initialPlacement>
           <territory>Western USA</territory>
           <quantity>2</quantity>
           <owner>British</owner>
        </initialPlacement>
        <initialPlacement>
           <territory>Eastern USA</territory>
           <quantity>10</quantity>
        </initialPlacement>
      </initialPlacements>
   </unit>

   <unit>
      <name>Stuka</name>
      <costs>
         <cost>
             <resource>PU</resource>
             <quantity>8</quantity>
         </cost>
         <cost>
             <resource>Oil</resource>
             <quantity>1</quantity>
         </cost>
       </costs>
      <availableTo>
         <player>Germany</player>
      </availableTo>
      <image>stuka.png</image>
      <initialPlacements>
        <initialPlacement>
           <territory>Southern Europe</territory>
           <quantity>2</quantity>
        </initialPlacement>
        <initialPlacement>
           <territory>Germany</territory>
           <quantity>10</quantity>
        </initialPlacement>
      </initialPlacements>
   </unit>
</unit>

[LaFayette]

Actually, the above example is not fair to XML as I wrote out the XML in it's most verbose form. If we condense it using attributes, it gets a lot nicer:

<units>

   <unit>
      <name>Infantry</name>
      <costs><cost resource="PU" quantity="2"/></costs>
      <image>infantry.png</image>
      <initialPlacements>
        <initialPlacement territory="Western USA" quantity="2" />
        <initialPlacement territory="Western USA" quantity="2" owner="British" />
        <initialPlacement territory="Eastern USA" quantity="2"/>
      </initialPlacements>
   </unit>

   <unit>
      <name>Stuka</name>
      <costs>
         <cost resource="PU" quantity="8"/>
         <cost resource="Oil" quantity="1"/>
      </costs>
      <availableTo>
        <player>Germany</player>
      </availableTo>
      <image>stuka.png</image>
      <initialPlacements>
        <initialPlacement territory="Southern Europe" quantity="2" />
        <initialPlacement territory="Germany" quantity="10" />
      </initialPlacements>
   </unit>
</unit>

There is a con list for us if we adopt YAML:

• we'll have two different major specification types
  • introduces complexity in the engine
  • not everyone will be able to give as much help while we learn YAML
  • we'll always be asking which spec someone is using, whether XML or YAML
• there will be complaints when features are available in one format and not another
• there is a cost to migrate XML files to YAML and we'll be doing that for a while
• there will be errors with indentations (potentially painful in large files)
• generally there is a learning curve for people (map makers) to switch

IMO the real benefit we get going forward is being able to write XML blocks that look like the above rather than what we have today. YAML and XML are equivalent, if we keep the XML to as nice as form as possible, the benefits of YAML become more marginal. We also would save a lot of work just by sticking to XML. It will also be likely painful when there are feature differences when YAML gets the newest and XML is left to stagnate.

I'm starting to think that if this were a fresh project, YAML would be a good contender, but the indentation problem would need to be seriously considered. For example, if a map has tabs and spaces in it, you then get into true hell as the indentation could look right, but would not be.

Therefore, IMHO I'm starting to land that unless we have a converter app to get everything into YAML then it would be a stronger consideration. If we can't cleanly migrate, I kinda think it'll just be the start of a lot of headaches for the benefit of a nicer syntax. Meanwhile the real problem is that the XML we have defined is not expressive and we can fix that without switching to YAML.

[Cernel]

I advise against moving around stuff, like that thing of having the cost in the "units" attachment, instead of in the production rules. In the example, I'm not seeing how that would allow players having different costs for the same unit.

I'm in favour, or at least not against, this general change only as long as everything stays as it is, beside changing the format (then, of course, once the new format is made and stable, specific changes can be considered).

[wc_sumpton]

@Cernel said in Should we support YAML for Maps?:

I advise against moving around stuff, like that thing of having the cost in the "units" attachment, instead of in the production rules. In the example, I'm not seeing how that would allow players having different costs for the same unit.

That is the exact problem the we have now. How do we change hitPoints for a unit for a player. What @LaFayette is advocating is that everything for a given item, territories (or 'zones'), and units be located in one place in the xml (or YAML). And which language to use, xml or YAML, to generate the map.

@LaFayette said in Should we support YAML for Maps?:

Meanwhile the real problem is that the XML we have defined is not expressive and we can fix that without switching to YAML.

I think this line says it all IMHO.

Cheers...

[Cernel]

@wc_sumpton That's hardly related. The unit's cost is not a charateristic of the unit itself. It is the mean, or should I say one of the means, by which the unit is placed on the "gameboard". It is more related to whatever trigger placing one or more units on the gameboard than it is to the options these units have in their attachments.

To make an example. I can make 1 single production rule saying that you spend 5 PUs plus 3 Mana plus 125 Bananas and receive 8 PUs plus 1 tech token plus 3 infantry units plus 15 dragon units. The PUs and the tech token are added to your resources stock, while the infantry and the dragon units are added to your inventory (to be likely, but not necessarily, eventually placed on the gameboard). There is no way you can say how many bananas an infantry cost, because you are using them, together with other resources, to buy both infantry and dragon units.

If this won't be anymore possible in the new format, then I oppose this format, as well as opposing if it is not absolutely assured that

everything currently possible will be still possible and loading the game from the game file (YAML) will not be slower

.

I'm not against removing possibilities. But this should be considered individually, not by going for a new and different system that we are not sure whether or not it is supporting in full what it is currently possible.

[ff03k64]

@Cernel said in Should we support YAML for Maps?:

I can make 1 single production rule saying that you spend 5 PUs plus 3 Mana plus 125 Bananas and receive 8 PUs plus 1 tech token plus 3 infantry units plus 15 dragon units.

125 bananas and you only get 3 infantry and 15 dragons? That is way over priced!

[LaFayette]

@Cernel The XML is there as an example to demonstrate what it would look like potentially, it's meant to be a realistic future example that is apples-to-apples between XML and YAML. I literally spent less 5 minutes on it, the focus is on YAML vs XML.

I can't help myself, it'd be easy to make it player specific:

      <costs>
         <cost player="Russia">
             <resource>PU</resource>
             <quantity>2</quantity>
         </cost>
         <cost player="Germany">
             <resource>PU</resource>
             <quantity>3</quantity>
         </cost>
      </costs>

[LaFayette]

@Cernel said in Should we support YAML for Maps?:

If this won't be anymore possible in the new format, then I oppose this format, as well as opposing if it is not absolutely assured that

I think there is a misunderstanding here. XML and YAML are essentially equivalent. The structure is going to change, but we'll still support the old structure for the sake of not breaking maps. The question is whether to go forward with an updated structure in XML or in YAML.

[LaFayette]

@Cernel said in Should we support YAML for Maps?:

everything currently possible will be still possible and loading the game from the game file (YAML) will not be slower

About this, yeah, we're not going to break maps. I don't think we'll have make any changes again where old maps are required to be updated. That has been a disaster every time we do it, we still have issues with maps using "attatchment" years after we "fixed" that.

Will updates to parsing be slower? Perhaps, it might very well be that your average map instead of loading in 600ms might load in 800ms or even 1.1s. Ideally we'll maintain parsing speed, but we've got issues with some maps crashing on parse due to memory and not because of speed. Going forward it's more important that we use less memory so that maps don't crash on parse due to OOM and we can restore XSS back to default which I suspect has caused further memory issues (particularly in bots).

War of Relics is a good benchmark, but it's an example where the XML structure we have is not expressive. Eventually if you are forced to copy/paste things too many times (my understanding is a macro was written to generate that as IIRC it's over 50k lines long), eventually you hit a limit.. So in part, the goal of more expressive XML is so you don't have to generate such macros and can nest things more properly so you don't have duplicate tags repeatedly to apply them to multiple entities.

[Black_Elk]

Hehe I enjoy tripleA for making me feel slightly less clueless on some stuff, but I'd have no clue on this one. Only time I've ever opened an xml file is to mess with this game, only time I've seen a yaml file was to play master of orion lol. Which approach do you think best La Fayette?

For someone who doesn't know thing one, the YAML postings certainly seem easier on the eyes haha. Wish I knew more though to have an opinion. Thumbs up though to whatever works!
Best Elk

[RogerCooper]

I always dislike relying on indentation for structures. I prefer explicit structural elements.