Map to Engine Compatibility Problems

Cernel

An example I remember, is that @RoiEX wanted to change a single misspelled name in the World At War map. If you would do something like that right now, you would make a lot of people more and more unable to complete savegames and cause a lot of crashes in the lobby and confusion in the general PBEM and whatever community.
And if people complains and you would, then, say, there was a "f" we changed into a "v" in a name, they would be like what the actual...?
But it is really not a big deal abstaining from making any compatibility breaking changes on popular maps, I think.

LaFayette

@cernel said in Map to Engine Compatibility Problems:

But it is really not a big deal abstaining from making any compatibility breaking changes on popular maps, I think.

I think this is where it dovetails with greater compatibility concerns. While I agree with this point, it does have an implications that we have a 'read-only' map that we are not changing as it's active and has saves. It makes sense to then create variants.

There is an implication here, the XML is controlled by a DTD file which constrains teh defintion of what can be in teh XML and the structure. We also have restrictions from the game engine of what it can read in XML. For me this is where the biggest problem comes in, we are left not changing maps (good), but still maintaining their XML anyways and occasionally breaking them due to game engine and DTD.

So if it were the case that the game engine could easily 'forever-read' immutable maps, then it would not be a problem at all. As is our maps are mutable, and the engine can't always read future and present maps.

LaFayette

@redrum said in another way, I think the problem is also that we don't have a maxEngine version tag : )
This is why I think so much of the trick is to make sure we don't lose compatibility to read existing maps.

redrum

@lafayette Uh. How would you use a maxEngine version tag?

RoiEX

@LaFayette About the territory naming problem:
Wouldn't it make sense then to simply give territories unique ids and use that internally instead of names as identifiers and keep names a visual-only thing?

CrazyG

@roiex
Great idea.

I currently have a map which I intend to use for multiple different time periods, and might rename areas with era-appropriate names. If I could use the same name within the centers, place, and polygon files for all iterations, but just had an XML option that changed the display name, it would be incredibly convenient. It would be great just for updating names in general, rather than change a minimum of four files, you just update the XML.

redrum

@roiex Yeah, I tend to agree with something along those lines to at least resolve the renaming issue but that would potentially require a lot of changes I believe. It also would only really solve the territory renaming challenge not other major map changes (though I think that is probably the most common case).

RoiEX

@redrum Perhaps the best choice would be to completely redesign maps using another format like yaml (because of the built-in types like arrays and objects/prototypes/maps/tables/whateveryouwanttocallit with modularity and shared map resources in mind.
This would of course require a lot of work and in depth brainstorming first, but such a brand new system could be the solution for a lot of problems we currently have.

redrum

@roiex So in an ideal world, I think that would probably be the right approach. The challenges that I see with doing that are:

Would we convert ALL existing maps to use the new yaml standard? That would have to include official, unofficial, and in-progress ones as much as possible and would probably be pretty painful. Alternatively, if we don't do that then we essentially have to support both xml and yaml standards which would be more maintenance and hard to provide map makers guidance.
Existing map makers would have to learn a completely new standard to utilize the new yaml system.

Given those 2 things, I lean towards trying to improve what we have over creating a new standard. I also think that while yaml would help in some places, I'm not convinced it would solve things like changing base map tiles. Some things like that would be hard to ever make compatible as you really have a different underlying map/territories at that point.

I think maybe we need an exercise in making the problems we want to consider a bit more concise. Do we really care about just renaming? Do we want to be able to change base map tiles? Change image names? To understand what are all the things that break map compatibility today and which items each potential solution would/could address.

RoiEX

@redrum If we actually manage to achieve such a new standard, I'd be happy to create a converter tool (perhaps javascript, on the website) that takes a map and converts it to the newer standard
I think having map makers to deal with change is a sacrifice we have to make ^^
Especially because the basic system of what a map needs to specify, turn oder, units, positions etc. will stay, just the way it's done will change.

redrum

@roiex Yeah, some form of converter tool would probably be necessary. I definitely would want mapmakers to be part of the process if we went down the road of a new standard map format. The last thing we want is a great new standard and all our map makers leave cause they hate it

Cernel

@roiex I feel scared by this proposal. Definitely would suggest supporting both standards indefinitely; if that is not feasible, maybe better not.

RoiEX

@Cernel Change is always scary, however there's no need to be worried.
Supporting both options defeats the purpose of redesigning the system IMO, so that's not an option.
However currently all of this is just some theoretical thinking.
There aren't any concrete plans yet, and even if there were, it would probably take months until everything works as expected.

RoiEX

@Cernel Also using an older version of the engine will always be an option ^^

Cernel

@roiex I'm mainly worried about everything in term of triggers changing raw elements of the game and any kind of advanced mapmaking stuff still working and the difficulty about doing such stuff if everything would become less intuitive?
Would a yaml standard allow faster loading and lower RAM requirement of heavy xml?

Hepps

@redrum Cost benefit analysis is a bitch with so many uncertain variables.

But I will say this...

Map makers tend to be more adaptive to change than users (broad brushing for sure).
The negative side effects of change can be mitigated if there are the proper resources available for people to acclimatize themselves to new processes and procedures.
As has been demonstrated in the past.... the forum and "monkey see-monkey do" have always been the benchmark for ways to use and change to new idea's and concepts. As long as there is a detailed functioning example (much like POS2)... and people to ask on the forum/chat.... obstacles can be over-come.

The key is really that it works right.... all the time... before it is launched and changes the community. and more than 1 or 2 people should be fluent in the new methods and procedures.

RoiEX

@Cernel YAML has the benefit of being well-readable and producing smaller file sizes than xml.
A good example of this is the official yaml website. This site is valid YAML and you can read the structure almost as normal text.
The probably biggest difference to xml is that while xml has tags with attributes that can be nested in each other, yaml consists solely of lists of elements (where the list is an element itself and can therefore be nested in itself) and sets of key-value pairs where the values can be a key-value pair set on its own or a list, or a string, or a number, or a boolean

LaFayette

@roiex said in Map to Engine Compatibility Problems:

Supporting both options defeats the purpose of redesigning the system IMO, so that's not an option.

I would respectfully disagree oh this. We can very easily partition the existing game parsing code and keep it working. Meanwhile adjustments can be done as version flags or whole new forks.

@RoiEX @redrum I'm really trying to convey and convince you that mass updates are not feasible (the evidence is against them, they were painful, another data point, Roger is still upgrading maps from 1.8 to 1.9!!! https://forums.triplea-game.org/topic/120/conversions-to-version-1-9

LaFayette

@hepps said in Map to Engine Compatibility Problems:

Map makers tend to be more adaptive to change than users (broad brushing for sure).

I've great respect for the level of expertise of map makers, frankly it requires way too much and should be much easier to churn out maps.

The subtle problem is that 95% of maps created are not active, we have 140 map repositories currently (a couple of those are admin/script, so around 137~138 maps!)

When a few maps want a new tag that breaks DTD; it's really quite the design flaw that we have to coordinate update of all 138 maps at the same time, get Roger involved, and we interfer with people being able to upgrade engine and start new games with updated maps.

RoiEX

@LaFayette If you believe legacy compatibility is a concern, I'd suggest going for an on-the-fly converter that would convert maps to a potential new format when needed.
This way we'd have the freedom to rework internal parsing code freely without having to be backwards compatible, but we could easily convert official maps to make our lives easier.