Categories and/or tags

[General_Zod]

@alkexr

Would be very awesome if you could share the script once completed. So others can utilize it.

[redrum]

So I've decided to resurrect this thread and take a stab at implementing some of this given some of the pain I've experienced updating the TWW XML.

@alkexr I know you ended up doing solution #3 and posted a few things around how it works:
https://forums.triplea-game.org/topic/888/middle-earth-battle-for-arda-official-thread/67
https://forums.triplea-game.org/topic/467/fallen-empire/88

It appears there are a couple areas that you tackled from what I can tell:

1. Allowing lists of elements (looks like you ended up using tags to do it)
2. Allowing a foreach loop for generating multiple attachments for a list of say players
3. Having a set of operations for combining lists: operations. ˇ is "or", or join; ^ is "and", or intersection, ~ is "and not"

Looking to see if you have any input/thoughts before I dive into this. I probably will focus on #1 first as it shouldn't be that hard. It appears you went with implementing a version of solution #2 around tags rather than solution #1 with categories, any reasoning there? I'm actually leaning more towards solution #1 as I think its probably more explicit and can manage all the lists in a single place.

[alkexr]

@redrum To me it was more straightforward to keep the definition of a unit in one place in the xml. The reason for this is probably found in the types of challenges I faced (having many-many unit types with many properties, and regularly having to tweak those for balance). This way you just have to look at the unit attachment and see all the "armor2"-s and "shield1"-s (or was it "shield2"? did it have "spearwall3"?) and whatnot, instead of having to look up categories far away in the xml. But looking back at the TWW example I provided above, I think that's clearly a case where categories are better.

The two systems don't have to be exclusive, though. I mean, why not have categories and something like <addToCategory name="anyTank"/>? These thingies could function similarly to <tag name="anyTank"/>, only changing in name to make it clear that this is the same system. But anyway, either is good I guess.

As for the operators, they are probably a worse choice than & and | and maybe !, but not having made up my mind by that point I was saving these characters for other potential future features for TagX.

Not sure if I have shared the TagX code yet, here it is in case it's helpful. It's actually remarkably good quality compared to my usual spaghetti.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using ConfigurationParser;
using System.Xml;
using System.IO;

namespace TAGX
{
    class Program
    {
        static List<string> log;
        static Dictionary<string, HashSet<string>> taglist;

        static void PreProcessNode(XmlNode node)
        {
            //Create HashSets based on tags, and remove tags from XML
            List<XmlNode> toberemoved = new List<XmlNode>();
            if (node.Attributes != null)
            {
                if (node.Attributes["attachTo"] != null)
                {
                    string nodename = node.Attributes["attachTo"].Value;
                    foreach (XmlNode subnode in node.ChildNodes)
                    {
                        if (subnode.Name == "tag")
                        {
                            if (subnode.Attributes["name"] == null)
                            {
                                log.Add("ERROR: tags must have a 'name' attribute.");
                                File.WriteAllLines("log.txt", log);
                                throw new Exception("Invalid tagx xml file.");
                            }
                            if (!taglist.ContainsKey(subnode.Attributes["name"].Value))
                                taglist.Add(subnode.Attributes["name"].Value, new HashSet<string>());
                            taglist[subnode.Attributes["name"].Value].Add(nodename);
                            //node.RemoveChild(subnode);
                            toberemoved.Add(subnode);
                        }
                    }
                }
            }

            foreach (XmlNode tbr in toberemoved)
                node.RemoveChild(tbr);

            foreach (XmlNode subnode in node.ChildNodes)
            {
                PreProcessNode(subnode);
            }
        }

        static HashSet<string> EvaluateExpression(string s, string logexpression)
        {
            if (!(s.Contains('(') || s.Contains(')') || s.Contains('ˇ') || s.Contains('~') || s.Contains('^')))
            {
                if (!taglist.ContainsKey(s) || taglist[s].Count == 0)
                {
                    log.Add("WARNING: nothing has the tag @" + s + "@. This can produce invalid game xml.");
                }
                return taglist[s];
            }
            int depth = 0;
            for (int i = 0; i < s.Length; i++)
            {
                if (s[i] == '(')
                {
                    depth++;
                    continue;
                }
                if (s[i] == ')')
                {
                    depth--;
                    if (depth < 0)
                    {
                        log.Add("ERROR: invalid parenthesis ')' at " + logexpression);
                        File.WriteAllLines("log.txt", log);
                        throw new Exception("Invalid tagx xml file.");
                    }
                    if (i == s.Length - 1)
                    {
                        if (depth > 0)
                        {
                            log.Add("ERROR: missing parenthesis ')' at " + logexpression);
                            File.WriteAllLines("log.txt", log);
                            throw new Exception("Invalid tagx xml file.");
                        }
                        else
                            return EvaluateExpression(s.Substring(1, s.Length - 2), logexpression); //we got to the end without incident - remove the parentheses
                    }
                    continue;
                }
                if (depth > 0)
                    continue; //we are within parentheses - don't do anything
                //here we already know that we are on the top level
                if (s[i] == 'ˇ')
                {
                    HashSet<string> arg1 = new HashSet<string>(EvaluateExpression(s.Substring(0, i), logexpression));
                    HashSet<string> arg2 = EvaluateExpression(s.Substring(i + 1), logexpression);
                    arg1.UnionWith(arg2);
                    return arg1; 
                }
                if (s[i] == '~')
                {
                    HashSet<string> arg1 = new HashSet<string>(EvaluateExpression(s.Substring(0, i), logexpression));
                    HashSet<string> arg2 = EvaluateExpression(s.Substring(i + 1), logexpression);
                    arg1.ExceptWith(arg2);
                    return arg1;
                }
                if (s[i] == '^')
                {
                    HashSet<string> arg1 = new HashSet<string>(EvaluateExpression(s.Substring(0, i), logexpression));
                    HashSet<string> arg2 = EvaluateExpression(s.Substring(i + 1), logexpression);
                    arg1.IntersectWith(arg2);
                    return arg1;
                }
            }
            //I dunno how we could potentially end up being here
            log.Add("ERROR: this expression defies the laws of logic: " + logexpression + " To avoid such incidents in the future please stop allowing your computer access to the Necronomicon.");
            File.WriteAllLines("log.txt", log);
            throw new Exception("Laws of logic have been proven useless.");
        }

        static void ReplaceTags(XmlNode node)
        {
            //Replace @tag calls@ with their respective HashSets as colon delimited lists
            if (node.Attributes != null)
                foreach (XmlNode attribute in node.Attributes)
                    ReplaceTags(attribute);
            foreach (XmlNode subnode in node)
            {
                if (subnode.NodeType == XmlNodeType.Text)
                {
                    if (subnode.Value.Count(c => c == '@') % 2 != 0)
                    {
                        log.Add("ERROR: invalid tag at >" + subnode.Value + "<. Use @tag@.");
                        File.WriteAllLines("log.txt", log);
                        throw new Exception("Invalid tagx xml file.");
                    }
                    while (subnode.Value.Contains('@'))
                    {
                        string before = subnode.Value.Substring(0, subnode.Value.IndexOf('@'));
                        string notbefore = subnode.Value.Substring(subnode.Value.IndexOf('@') + 1);
                        string between = notbefore.Substring(0, notbefore.IndexOf('@'));
                        string after = notbefore.Substring(notbefore.IndexOf('@') + 1);
                        /*if (!taglist.ContainsKey(between) || taglist[between].Count == 0)
                        {
                            log.Add("WARNING: nothing has the tag @" + between + "@. This can produce invalid game xml.");
                        }*/
                        string colondelimited = HashToColonDelimited(EvaluateExpression(between, between));
                        if (colondelimited == "")
                            log.Add("WARNING: nothing matches the expression @" + between + "@. This can produce invalid game xml.");
                        subnode.Value = before + colondelimited + after;
                    }
                }
                ReplaceTags(subnode);
            }
        }

        static string HashToColonDelimited(HashSet<string> hash)
        {
            string result = "";
            foreach (string s in hash)
                result += ':' + s;
            return result.Substring(1);
        }

        static void ReplaceForeach(XmlNode node)
        {
            List<XmlNode> tbr = new List<XmlNode>();
            List<XmlNode> tba = new List<XmlNode>();
            foreach (XmlNode child in node.ChildNodes)
            {
                tbr.Add(child);
                foreach (XmlNode foreached in EvaluateForeach(child))
                    tba.Add(foreached);
            }
            foreach (XmlNode r in tbr)
                node.RemoveChild(r);
            foreach (XmlNode a in tba)
                node.AppendChild(a);
            foreach (XmlNode child in node.ChildNodes)
                ReplaceForeach(child);
        }

        static List<XmlNode> EvaluateForeach(XmlNode node)
        {
            List<XmlNode> list = new List<XmlNode>();
            list.Add(node);
            return EvaluateForeach(list);
        }

        static List<XmlNode> EvaluateForeach(List<XmlNode> nodes)
        {
            //Return each node foreached properly, without respect to possible subnode foreaches
            List<XmlNode> result = new List<XmlNode>();
            foreach (XmlNode node in nodes)
            {
                if (node.Attributes != null)
                {
                    if (node.Attributes["foreach"] != null)
                    {
                        try
                        {
                            string command = node.Attributes["foreach"].Value;
                            string firstforeachcommand = command.Split(';')[0];
                            string key = firstforeachcommand.Split('=')[0];
                            List<string> values = firstforeachcommand.Split('=')[1].Split(':').ToList();
                            List<XmlNode> foreached = new List<XmlNode>();
                            foreach (string value in values)
                            {
                                XmlNode tba = node.Clone();
                                Substitute(tba, "$" + key + "$", value);
                                //remove the foreach we just completed, we don't want to do it again and again
                                if (command.Split(';').Length == 1)
                                    tba.Attributes.Remove(tba.Attributes["foreach"]);
                                else
                                    tba.Attributes["foreach"].Value = tba.Attributes["foreach"].Value.Remove(0, firstforeachcommand.Length + 1);
                                foreached.Add(tba);
                            }
                            foreach (XmlNode foreachednode in EvaluateForeach(foreached))
                                result.Add(foreachednode);
                        }
                        catch (Exception)
                        {
                            log.Add("ERROR: invalid foreach call at node >" + node.ToString() + "<.");
                            File.WriteAllLines("log.txt", log);
                            throw new Exception("Invalid tagx xml file.");
                        }
                    }
                    else
                        result.Add(node);
                }
                else
                    result.Add(node);
            }
            return result;
        }

        static void Substitute(XmlNode node, string key, string value)
        {
            if (node.Attributes != null)
                foreach (XmlNode attribute in node.Attributes)
                    Substitute(attribute, key, value);
            foreach (XmlNode subnode in node)
            {
                if (subnode.NodeType == XmlNodeType.Text)
                {
                    while (subnode.Value.Contains(key))
                    {
                        subnode.Value = subnode.Value.Replace(key, value);
                    }
                }
                Substitute(subnode, key, value);
            }
        }

        static void Main(string[] args)
        {
            log = new List<string>();
            log.Add("Base directory: " + AppDomain.CurrentDomain.BaseDirectory);

            XmlDocument xmlconfig;
            string filename = "";

            if (args == null || args.Length == 0)
            {
                log.Add("No configuration file provided. Default configuration file >TAGX.tagxcfg< will be used.");
                filename = AppDomain.CurrentDomain.BaseDirectory + "TAGX.tagxcfg";
            }
            else
            {
                filename = args[0];
                log.Add("Loading configuration file >" + filename + "<.");
            }
            if (File.Exists(filename))
            {
                try
                {
                    xmlconfig = new XmlDocument();
                    xmlconfig.Load(filename);
                }
                catch (Exception)
                {
                    log.Add("ERROR: could not load or parse configuration file >" + filename + "<.");
                    File.WriteAllLines("log.txt", log);
                    return;
                }
            }
            else
            {
                log.Add("ERROR: configuration file >" + filename + "< does not exist.");
                File.WriteAllLines("log.txt", log);
                return;
            }

            string inpath = "";
            string outpath = "";

            try
            {
                inpath = xmlconfig.SelectSingleNode("//cfg[@key='input xml path']").Attributes["value"].Value;
                outpath = xmlconfig.SelectSingleNode("//cfg[@key='output xml path']").Attributes["value"].Value;
            }
            catch (Exception)
            {
                log.Add("ERROR: configuration file is invalid. Make sure 'input xml path' and 'output xml path' are defined.");
                File.WriteAllLines("log.txt", log);
                return;
            }

            XmlDocument source = new XmlDocument();
            Environment.CurrentDirectory = AppDomain.CurrentDomain.BaseDirectory;
            try
            {
                source.Load(inpath);
            }
            catch (Exception e)
            {
                log.Add("Could not load xml from >" + inpath + "<: " + e.Message);
                File.WriteAllLines("log.txt", log);
                return;
            }

            taglist = new Dictionary<string, HashSet<string>>();

            try
            {
                PreProcessNode(source);
                ReplaceTags(source);
            }
            catch (Exception)
            {
                log.Add("Exception occured while substituting tags.");
                File.WriteAllLines("log.txt", log);
                return;
            }
            try
            {
                ReplaceForeach(source);
            }
            catch (Exception)
            {
                log.Add("Exception occured while evaluating foreaches.");
                File.WriteAllLines("log.txt", log);
                return;
            }

            log.Add("Processing xml completed successfully.");

            try
            {
                Environment.CurrentDirectory = AppDomain.CurrentDomain.BaseDirectory;
                source.Save(outpath);
            }
            catch (Exception)
            {
                log.Add("Could not save game xml.");
                File.WriteAllLines("log.txt", log);
                return;
            }
            log.Add("Game xml saved.");
            File.WriteAllLines("log.txt", log);
        }
    }
}

[redrum]

@alkexr Thanks for the input and posting your code. Yeah, its possible to implement both and even so you could use a combination like you point out. I think I'm gonna start with categories and see how that goes and then consider adding the ability to 'tag' to add to or create a category.

I think the next question is what's the best/easiest way to define them. I don't think I'm going to do any validation at the category level and just treat them as essentially a list of strings which get validated later on when they actually set into the values. Any thoughts on whether its easier to have each item as its own element vs just a colon delimited list? I'd also think if going with each item as its own element then using its base type is easier than a generic 'element'.

Also, do you think category is the best thing to call it? Couple alternatives I thought of is "list" or "variable" or "map".

Option #1

    <category name="anyMech.Infantry">
        <element name="germanMech.Infantry"/>
        <element name="italianMech.Infantry"/>
        <element name="japaneseMech.Infantry"/>
    </category>

Option #2

    <category name="anyMech.Infantry" value="germanMech.Infantry:italianMech.Infantry:japaneseMech.Infantry"/>

Option #3

    <category name="anyMech.Infantry">
        <unit name="germanMech.Infantry"/>
        <unit name="italianMech.Infantry"/>
        <unit name="japaneseMech.Infantry"/>
    </category>
    <category name="axisPlayers">
      <player name="Italians"/>
      <player name="Germans"/>
      <player name="Japanese"/>
    </category>

[alkexr]

@redrum I would call it a variable if there is a potential future plan to add / remove elements using triggers. Map would be a confusing term. List or category are both fine. But category feels wrong with #2.

#2 is only good if you have to write hundreds of these and length is a concern. I think that typically this part of xml editing is going to be negligible in terms of time spent typing, while clarity at a glance is probably more important here than anywhere.

[redrum]

@alkexr I like variable as well just wasn't sure if its too technical for most map makers.

Ok I'll go with something like option #1 or #3 then.

Do you think variables are useful in anything besides attachments?

[alkexr]

@redrum said in Categories and/or tags:

Do you think variables are useful in anything besides attachments?

Foreach loops could be used for turn sequences or production frontiers, maybe even unit placement (like placing an invisible factory on every territory - though defining ownership is a problem) or map options (like "Player1 is AI"). But staying within the realm of reason, these are the only possibilities I can think of. (Of course there are always some mapmakers who stray out of said realm sometimes ) Without foreach loops, they can only be used as colon delimited lists, which only happen in attachments at the moment.

[redrum]

Alright. So got some initial code working around tackling defining variables. Here in a example of TWW:

  <variableList>
    <variable name="AntiTankTargets">
      <element name="germanMech.Infantry"/>
      <element name="germanTank"/>
      <element name="germanHeavyTank"/>
      <element name="germanHeavyTank-damaged"/>
      <element name="russianMech.Infantry"/>
      <element name="russianTank"/>
      <element name="russianHeavyTank"/>
      <element name="russianHeavyTank-damaged"/>
      <element name="americanMech.Infantry"/>
      <element name="americanTank"/>
      <element name="americanHeavyTank"/>
      <element name="americanHeavyTank-damaged"/>
      <element name="italianMech.Infantry"/>
      <element name="italianTank"/>
      <element name="italianHeavyTank"/>
      <element name="italianHeavyTank-damaged"/>
      <element name="japaneseMech.Infantry"/>
      <element name="japaneseTank"/>
      <element name="japaneseHeavyTank"/>
      <element name="japaneseHeavyTank-damaged"/>
      <element name="britishMech.Infantry"/>
      <element name="britishTank"/>
      <element name="britishHeavyTank"/>
      <element name="britishHeavyTank-damaged"/>
      <element name="chineseMech.Infantry"/>
      <element name="chineseTank"/>
      <element name="spanishTank"/>
      <element name="brazilianTank"/>
      <element name="turkishTank"/>
      <element name="swedishTank"/>
    </variable>
  </variableList>

    <attachment name="unitAttachment" attachTo="germanAntiTankGun" javaClass="games.strategy.triplea.attachments.UnitAttachment" type="unitType">
      <option name="attack" value="1"/>
      <option name="defense" value="1"/>
      <option name="isAAforCombatOnly" value="true"/>
      <option name="typeAA" value="AntiTankGun"/>
      <option name="targetsAA" value="$AntiTankTargets$"/>
      <option name="maxAAattacks" value="1"/>
      <option name="maxRoundsAA" value="-1"/>	  
      <option name="transportCost" value="2"/>
      <option name="mayOverStackAA" value="true"/>
      <option name="movement" value="1"/>
      <option name="attackAA" value="2"/>
      <option name="isLandTransportable" value="true"/>
      <option name="requiresUnits" value="germanFactory"/>
      <option name="canBeGivenByTerritoryTo" value="Germany"/>
      <option name="damageableAA" value="true"/>
      <option name="canInvadeOnlyFrom" value="none"/>
    </attachment>

[redrum]

And an example of nested variables:

    <variable name="russianAntiAirTargets">
      <element name="russianFighter"/>
      <element name="russianTacticalBomber"/>
    </variable>
    <variable name="americanAntiAirTargets">
      <element name="americanFighter"/>
      <element name="americanTacticalBomber"/>
    </variable>
    <variable name="AntiAirTargets">
      <element name="russianAntiAirTargets"/>
      <element name="americanAntiAirTargets"/>
    </variable>

[Frostion]

This sounds great. I really think the “Solution #1: Categories” or similar would be a great feature, so that we don’t have to list all units. Awesome if triggers could change these codes!

I think the most intuitive way to give a unit anti-air targets would be the example:

<option name="targetsAA" value="anyMech.Infantry:anyTank:anyHeavyTank"/>

And then the lists being like

<category name="anyTank" type="unit">
        <element name="germanTank"/>
        <element name="italianTank"/>
        <element name="japaneseTank"/>
</category>

I think understanding the code would be could be even more intuitive for mapmakers if it was not </category> but instead </unitCategory> .

Maybe the above (unit)category code could also have a ”players” setting with a list of affected players or unaffected players, so that like some enemy units can evade this specific aa attack. Default could be ALL if not used. But this could open up for players having the same units with the same name, lige “Fighter” and “Bomber”. So like if a player develops “stealth” then his fighters/bombers can become immune from some attacks after the player is removed by trigger from the code. If a player develops “flares” then missiles can’t hit or reactive armor that disrupts RPG units’ if they have an AA attack. You get my meaning

I must admit that I don’t really understand the talk about “tags” either. Also, I don’t understand how “variables” can or should be understood in this context. To me it seems like language out of a mathematical equation.

[Cernel]

https://en.wikipedia.org/wiki/Variable_(mathematics)
a variable is a symbol, commonly a single letter, that represents a number, called the value of the variable, which is either arbitrary, not fully specified, or unknown.

Mutatis mutandis, something defined as "variable" inside the xml makes me think of something that is strictly numerical and it is not fixedly defined by the xml itself (that is or may be stored somewhere else). For example, something that the user may customize (for example, called from a potentially editable property).

[redrum]

@Frostion For now I'm leaning towards calling them 'variables' instead of 'categories'. 'unitCategory' wouldn't really work since you can have a list of player, territories, etc as well not just units. The easiest way to think about it is this is essentially a find/replace that's useful when you end up having the same list of 'elements' copy and pasted multiple times.

@Cernel This is the definition that would be more accurate: https://en.wikipedia.org/wiki/Variable_(computer_science)
In computer programming, a variable or scalar is a storage location (identified by a memory address) paired with an associated symbolic name (an identifier), which contains some known or unknown quantity of data referred to as a value. The variable name is the usual way to reference the stored value, in addition to referring to the variable itself, depending on the context. This separation of name and content allows the name to be used independently of the exact data it represents.

Essentially, they are key/value pairs where you define a variable 'name' and that represents the list of elements (units, players, territories, etc). Where ever you use that variable, it is replaced with the list of elements.

[Cernel]

@redrum That thing I'd rather name it a set.

https://en.wikipedia.org/wiki/Set_(mathematics)

[redrum]

@Cernel Actually if anything it would be a "list" not a "set" since you could in theory have duplicates of the same element assigned to the same one.

[Cernel]

@redrum List is too generic; the xml is a bunch of lists. If not "set", then "group".

What would be a user case for having something like:

<category name="anyMech.Infantry">
    <element name="germanMech.Infantry"/>
    <element name="italianMech.Infantry"/>        
    <element name="italianMech.Infantry"/>        
    <element name="japaneseMech.Infantry"/>
</category>

If none, then better that the program doesn't accept duplicates.

[redrum]

@Cernel I'm adding the foreach ability along with this so you could do something like this for an attachment where you have 2 "lists" of the same length that are looped through to create in this case 4 attachments:

<attachment foreach="anyMech.Infantry:anyMechPlayers"

<variable name="anyMech.Infantry">
    <element name="germanMech.Infantry"/>
    <element name="italianMech.Infantry"/>        
    <element name="italianMech.Infantry"/>        
    <element name="japaneseMech.Infantry"/>
</variable>

<variable name="anyMechPlayers">
    <element name="Germany"/>
    <element name="Italy"/>        
    <element name="ItalianMinors"/>        
    <element name="Japan"/>
</variable>

[Cernel]

@redrum Will "foreach" work for triggers (for example, allowing a trigger to fire once for each of some units in a territory?). If so, will this make "each" deprecate?

Back on the matter in case duplicates are not redundant (meaning "germanMech.Infantry+italianMech.Infantry+italianMech.Infantry+japaneseMech.Infantry" is not the same as "germanMech.Infantry+italianMech.Infantry+japaneseMech.Infantry"), it can be called "sequence".

https://en.wikipedia.org/wiki/Sequence

The only matter, at this point, is that in sequences the order is usually supposed to have significance.

I don't recall if there is a term for something repetitions sensitive but not positions sensitive (set isn't for both and sequence is for both).

[Cernel]

@Cernel said in Categories and/or tags:

I don't recall if there is a term for something repetitions sensitive but not positions sensitive

Apparently, it's called "multiset".

https://en.wikipedia.org/wiki/Multiset

[Frostion]

I really hope that the term is at least a word known to ordinary people and not technobabble gibberish

[Cernel]

@Frostion said in Categories and/or tags:

I really hope that the term is at least a word known to ordinary people and not technobabble gibberish

Knuth himself attributes the first study of multisets to the Indian mathematician Bhāskarāchārya, who described permutations of multisets around 1150. Knuth also lists other names that were proposed or used for this concept, including list, bunch, bag, heap, sample, weighted set, collection, and suite.

I just advice against list, as pretty much all the xml is a bunch of lists, so that may be confusing if you are talking about some lists.