Map to Engine Compatibility Problems

[RoiEX]

@LaFayette About the territory naming problem:
Wouldn't it make sense then to simply give territories unique ids and use that internally instead of names as identifiers and keep names a visual-only thing?

[CrazyG]

@roiex
Great idea.

I currently have a map which I intend to use for multiple different time periods, and might rename areas with era-appropriate names. If I could use the same name within the centers, place, and polygon files for all iterations, but just had an XML option that changed the display name, it would be incredibly convenient. It would be great just for updating names in general, rather than change a minimum of four files, you just update the XML.

[redrum]

@roiex Yeah, I tend to agree with something along those lines to at least resolve the renaming issue but that would potentially require a lot of changes I believe. It also would only really solve the territory renaming challenge not other major map changes (though I think that is probably the most common case).

[RoiEX]

@redrum Perhaps the best choice would be to completely redesign maps using another format like yaml (because of the built-in types like arrays and objects/prototypes/maps/tables/whateveryouwanttocallit with modularity and shared map resources in mind.
This would of course require a lot of work and in depth brainstorming first, but such a brand new system could be the solution for a lot of problems we currently have.

[redrum]

@roiex So in an ideal world, I think that would probably be the right approach. The challenges that I see with doing that are:

1. Would we convert ALL existing maps to use the new yaml standard? That would have to include official, unofficial, and in-progress ones as much as possible and would probably be pretty painful. Alternatively, if we don't do that then we essentially have to support both xml and yaml standards which would be more maintenance and hard to provide map makers guidance.
2. Existing map makers would have to learn a completely new standard to utilize the new yaml system.

Given those 2 things, I lean towards trying to improve what we have over creating a new standard. I also think that while yaml would help in some places, I'm not convinced it would solve things like changing base map tiles. Some things like that would be hard to ever make compatible as you really have a different underlying map/territories at that point.

I think maybe we need an exercise in making the problems we want to consider a bit more concise. Do we really care about just renaming? Do we want to be able to change base map tiles? Change image names? To understand what are all the things that break map compatibility today and which items each potential solution would/could address.

[RoiEX]

@redrum If we actually manage to achieve such a new standard, I'd be happy to create a converter tool (perhaps javascript, on the website) that takes a map and converts it to the newer standard
I think having map makers to deal with change is a sacrifice we have to make ^^
Especially because the basic system of what a map needs to specify, turn oder, units, positions etc. will stay, just the way it's done will change.

[redrum]

@roiex Yeah, some form of converter tool would probably be necessary. I definitely would want mapmakers to be part of the process if we went down the road of a new standard map format. The last thing we want is a great new standard and all our map makers leave cause they hate it

[Cernel]

@roiex I feel scared by this proposal. Definitely would suggest supporting both standards indefinitely; if that is not feasible, maybe better not.

[RoiEX]

@Cernel Change is always scary, however there's no need to be worried.
Supporting both options defeats the purpose of redesigning the system IMO, so that's not an option.
However currently all of this is just some theoretical thinking.
There aren't any concrete plans yet, and even if there were, it would probably take months until everything works as expected.

[RoiEX]

@Cernel Also using an older version of the engine will always be an option ^^

[Cernel]

@roiex I'm mainly worried about everything in term of triggers changing raw elements of the game and any kind of advanced mapmaking stuff still working and the difficulty about doing such stuff if everything would become less intuitive?
Would a yaml standard allow faster loading and lower RAM requirement of heavy xml?

[Hepps]

@redrum Cost benefit analysis is a bitch with so many uncertain variables.

But I will say this...

1. Map makers tend to be more adaptive to change than users (broad brushing for sure).

2. The negative side effects of change can be mitigated if there are the proper resources available for people to acclimatize themselves to new processes and procedures.

3. As has been demonstrated in the past.... the forum and "monkey see-monkey do" have always been the benchmark for ways to use and change to new idea's and concepts. As long as there is a detailed functioning example (much like POS2)... and people to ask on the forum/chat.... obstacles can be over-come.

The key is really that it works right.... all the time... before it is launched and changes the community. and more than 1 or 2 people should be fluent in the new methods and procedures.

[RoiEX]

@Cernel YAML has the benefit of being well-readable and producing smaller file sizes than xml.
A good example of this is the official yaml website. This site is valid YAML and you can read the structure almost as normal text.
The probably biggest difference to xml is that while xml has tags with attributes that can be nested in each other, yaml consists solely of lists of elements (where the list is an element itself and can therefore be nested in itself) and sets of key-value pairs where the values can be a key-value pair set on its own or a list, or a string, or a number, or a boolean

[LaFayette]

@roiex said in Map to Engine Compatibility Problems:

Supporting both options defeats the purpose of redesigning the system IMO, so that's not an option.

I would respectfully disagree oh this. We can very easily partition the existing game parsing code and keep it working. Meanwhile adjustments can be done as version flags or whole new forks.

@RoiEX @redrum I'm really trying to convey and convince you that mass updates are not feasible (the evidence is against them, they were painful, another data point, Roger is still upgrading maps from 1.8 to 1.9!!! https://forums.triplea-game.org/topic/120/conversions-to-version-1-9

[LaFayette]

@hepps said in Map to Engine Compatibility Problems:

Map makers tend to be more adaptive to change than users (broad brushing for sure).

I've great respect for the level of expertise of map makers, frankly it requires way too much and should be much easier to churn out maps.

The subtle problem is that 95% of maps created are not active, we have 140 map repositories currently (a couple of those are admin/script, so around 137~138 maps!)

When a few maps want a new tag that breaks DTD; it's really quite the design flaw that we have to coordinate update of all 138 maps at the same time, get Roger involved, and we interfer with people being able to upgrade engine and start new games with updated maps.

[RoiEX]

@LaFayette If you believe legacy compatibility is a concern, I'd suggest going for an on-the-fly converter that would convert maps to a potential new format when needed.
This way we'd have the freedom to rework internal parsing code freely without having to be backwards compatible, but we could easily convert official maps to make our lives easier.

[RoiEX]

@LaFayette To clarify, a new format would have map variations in mind and clearly seperate resources and logic to allow us to update maps individually.

[LaFayette]

@RoiEX an on-the-fly converter is intriguing for a few uses.
I think it make sense to use that to generate new, updated versions of maps.

But, on another level, if we have such a converter, why not just ship it with the game engine? Would it be feasible for example to create one so 1.9 engines could read 1.8 maps again?

[LaFayette]

@cernel said in Map to Engine Compatibility Problems:

Would a yaml standard allow faster loading and lower RAM requirement of heavy xml?

It is CPU requirement for heavy XML : )
That is a problem.
]
YAML could help as it can be more expressive than XML (easier to read too). If we do it right, we can drastically cut down on looping while reading files. I think personally YAML would dovetail with a possible future save game native format, so we could have one file hold more information in one file.

I tried an experiment of converting a map to YAML, it does make it a bit cleaner, but does not radically cut down on complexity for a simple example: https://github.com/DanVanAtta/triplea/commit/5e1ae21c7e16ae952a9ad9cbb95ce75155c44937#diff-0dff504360f8239d76828118a02ebadd (that example is the Test.xml file converted to yaml, for comparison the original XML version is: https://github.com/triplea-game/triplea/blob/master/game-core/src/test/resources/Test.xml)

[RoiEX]

@LaFayette I was talking about shipping a converter with the engine, but I wanted to point out the converting mechanism, so the original map file is never being read by the game engine, but converted into an actual newer file format (and deleting the old map file?).
About YAML saving: Ideally we'd make use of built-in data structures of YAML which will save a lot of space, meaning for example no attachment, but properties instead that can be set or default to a default value and will make a lot of details implicit rather than explicitl.
Example:

units:
 - name: test
   water: true

etc.