RFC: Map Rating to Replace Map Categories

LaFayette

To clarify a bit on the transition,
best -> 4
good -> 3
experimental -> 1

Out of the experimental, those that are not broken would be a '2'. If they turn out to be balanced and a good map then they move to a '3'.

Hence, '4' would be the all-star list. '3' is the list of very good maps, but they do not like nice, but certainly would be great to play. '2' is something that only should be played if you are actively trying to balance the map, otherwise avoid it. '1' is just to be avoided unless trying to help build a map or get some early feedback.

If we change our perspective that maps are never to be uploaded until done, then '1' and '2' mostly go away and then we are back to only having the sneaker-net solutions that many have developed over time (which is again frustrating since in many ways it is far more work for everyone then simply uploading to github (which is easier than uploading to dropbox even)).

So clearly I think it's beneficial for the rating system to also take into account maps that are not yet done, those that are newly done, and then the vast majority of maps that are done and has some philosophy on how to distinguish them so we are not left with a list of 70+ maps.

Cernel

The fact that the mention of map-skins was made between parentheses should have hinted that was unnecessary: I don't see how the value of a game and the quality of a skin for playing the game can be mixed together and I believe they ought not to even if every map would have only one game and every game would have only one skin (both being not the case).

Substantially, based on what you said, your 1, 2 and 3 categories would rate the game only, whereas the value 4 would be a 3 for the game and something not too low a rating for the graphic (and also the sounds set or not?).

All this, of course, assuming that one wants to discriminate maps based on a rating system (I'm not saying anything about it because I understand this topic implies that rating has to be the way.).

LaFayette

@Cernel Do you have any suggestions on how we can avoid the majority of all maps from having the exact same rating? If we delete the experimental maps, then presumably without any other factor all maps would have the same rating. This fails to solve problem (C) in the OP.

LaFayette

@Cernel said in RFC: Map Rating to Replace Map Categories:

I'm not saying anything about it because I understand this topic implies that rating has to be the way.).

Solving the 3 problems in the OP is the goal. How we do that is an open question.

This rating system suggestion is essentially the existing categories remapped to numbers with a definition between '1' and '2' (both considered experimental today) and a definition between '3' (good) and '4' (best).

If there is a better way to solve the 3 problems without incurring continuous overhead, then kindly offer it.

Cernel

@LaFayette said in RFC: Map Rating to Replace Map Categories:

@Cernel Do you have any suggestions on how we can avoid the majority of all maps from having the exact same rating? If we delete the experimental maps, then presumably without any other factor all maps would have the same rating. This fails to solve problem (C) in the OP.

I think this can be seriously solved only by having something collecting popularity data of some sort and returning information about what are the most played maps. Besides, I'm fairly certain the current divide between the highest and second highest category in the current download listing is not based on graphics (there are certainly many maps in the second level of quality which are about as good looking as what you can find in the first level), so I'm sorry to say you would be quite wrong in putting all first level maps on the 4 rating and all second level maps on the 3 rating under what described (Of course, this is merely a legacy problem given by the fact that I don't believe there is any kind of information about what currently distinguish the first and the second level.).

Of course, I realize this is substantially off-topic, so no: I don't have any suggestions on how to solve such challenge within a rating system. After a level of dencent quality, it is arguably just too subjective for a no-profit organization to be as judgmental as needed.

LaFayette

@Cernel Map popularity is not happening in 2.6 and triplea_maps.yaml is disappearing. Seemingly this must be solved or we can say we are unable to solve this. Furthermore, I don't agree that popularity solves (C). Popularity could be gamed, for example someone just repeatedly starts a map over and over again. You could also just leave a specific open for a very long time, create many alias accounts (emails are infinite) and then start a game scenario many times. Last, popularity biases towards older maps, that's another challenge to solve that also fails (C).

We could just keep all the same categories, but that feels like a shame since they are completely subjective and are really just whatever Veq felt they should have been some 12 years ago.

I'm sorry to say you would be quite wrong in putting all first level maps on the 4 rating and all second level maps on the 3 rating under what described

Yip, so there would be some shifting of some maps to 3 and others up to 4. Now that we have a working definition of what is a 3 vs 4, we can actually move stuff without being completely subjective or just leaving it up to me to make an executive decision. If I were to make executive decision, then a year from now we'll be in the same place where 'best' is whatever I felt it should be and no way to shift maps between categories.

Of course, we can work on the rating definitions, for example maybe somehow 5 levels would make more sense. Maybe the difference between 3 and 4 could be clarified better. Maybe a different system that does not involve significant work (now and in an ongoing basis) could be suggested that also fits with the map development workflow.

IDK, my best idea for now is to give 'good' and 'best' definitions and to split 'experimental' between broken and in-progress.

Cernel

@LaFayette I would then have all 3 and 4 in 3 even if this means giving up with having a category limited to the very good maps.

What I think I made clear is that I believe it just makes no sense to have on the same metre two different things. The quality of the game is one thing and the quality of the graphic (and sounds?) is another. We cannot say what is better between a game which is very engaging but has very bad graphics and a game which is very boring but has very good graphics.

If keeping the current concept, I suggest changing the name of "4" to "3+". This way, you could say that "3" and "3+" maps are about the same quality of gaming (for at least one of their games) but "3+" have a particularly good graphic (and sounds?) too.

split 'experimental' between broken and in-progress.

I agree, but be ready having the category for not-yet-good maps which are actively under work be empty most of the time and with some significant over-head for assuring abandoned maps won't linger in there. Still, this category may be very good to have when there is actually a map in it (to avoid burying it within all abandoned not-very-good ones). I'm just saying it will be important resisting to the temptation of removing it once it will (because it will) remain completely empty for a long time.

LaFayette

@Cernel said in RFC: Map Rating to Replace Map Categories:

The quality of the game is one thing and the quality of the graphic (and sounds?) is another. We cannot say what is better between a game which is very engaging but has very bad graphics and a game which is very boring but has very good graphics.

I hear ya. The rating suggestion is not just a ranking, think of it more as levels.

Is the game complete? -> 2
Is the game good and balanced? -> 3
Is the game well polished and looks good? -> 4

A map cannot be a rating (4) without being good, balanced, and complete. Does this make sense and do you think it would work?

I agree, but be ready having the category for not-yet-good maps which are actively under work be empty most of the time and with some significant over-head for assuring abandoned maps won't linger in there.

Indeed, and if we purge the broken maps then almost all maps are going to be a 3 or a 4. That I think is okay, the '4' are then the answer to (C) in the OP. Otherwise players should look through and download all the others as they are fine maps.

Cernel

There are actually some topics currently open in forum to gather popularity information:
https://forums.triplea-game.org/topic/2394/triplea-vote-most-popular-map
https://forums.triplea-game.org/topic/1234/what-games-are-you-all-playing

If the forum would accept polls with huge numbers of options, one could have a poll with one choice per map or game.

If anyone were to take upon himself or herself the difficult duty of indicating the very good maps amongst the good ones (as we guess Veqryn used somehow to do), he or she could do it by gathering the votes. Otherwise, I'm still oriented suggesting having only your first 3 levels and maybe a 3+ for the level 3 games which happen to have an original skin which is very good (if not a 1-3 rating for the game (the best one if the map has more than one) and a 1-3 rating for the skin).

Also, I would have 1 as the best and higher numbers the worst.

LaFayette

I was planning to replace the rating number with a star count.

I think it's okay to open up this conversation in retrospect, even if all elements of what we design are not onboarded in 2.6 at least we will have the roadmap and features that are more likely to fit together.

I think the three main 'attributes' (aka tags) on maps are:

rating
era
popularity

For rating we need to define definitions. I'm not clear why "3+ should not just be called "4"? I do think that whatever the best skin is, that should be default, beyond that, we ignore skins when it comes to this.

Era would take some work to define some good categories for it. I don't think it would be helpful to have as many era's as we do maps, or to have all maps be in their unique era and then a big mass in WWII.

If a map maker typo's an era and defines one that does not exist, how should that be handled? Does that get created as a new ear, or does the map appear as having no defined era?

For popularity, I think we need to be sure it stays sustainable. It may get a lot of excitement and action now, but will it 5 years from now? I think to this extent it probably needs to be automated. I also think we need to develop a popularity score that is applied to all maps. If we define the top 3 most popular, I don't think that is necessarily useful or justice that there are so many maps and there are differences in popularity when get to the long tail (for example, Cold War once used to be popular, Napolean used to be extremely popular and is less so today. Having those not be somewhere in the middle or top middle of the list seems inadequate).

ubernaut

@LaFayette so if the premise here is that our current categories really signal quality then I would agree in a general sense, at least. might be good to include tags as well in that case.

LaFayette

In essence I think rating, era, and popularity will essentially be implemented as tags that are produced from the maps server. Rather than having completely free-form tags, I'm leaning for the tag keys and values to be on a white list and perhaps ignored if not on that list. I'm still not sure if we can't avoid a lot of pain for typo's or issues if we want to redo or change the era tag names.

TheDog

I would like to keep the experimental rating of 1 and display it for download. It has a lot of resources that can be plundered for maps, units, flags and ideas, I would be sad to see it go.

Also to incorporate a played often at some point in the maps life.

I like the star rating as per LaFayette with;
1-Incomplete (old Experimental)
2-Complete
3-Complete & Balanced
4-As 3, Looks good or is/has played often
5-As 3 Looks good & is/has played often, best in its era (can have more than 1 5star per era )

.
More on the era
As an example the map maker assigns era code, 3-4 letter/number code, white list checked, if the map maker gets it wrong it is changed to WRON=wrong.
This code is then displayed in English for the public to use for downloading.
eg WW2W=1939-1945 WW2 World theatre

Note that WW2 has been split into 3 theatres

FANT=Fantasy
ANCI=3000bce to 1450 Ancient/Medieval
RENA=1450-1793 Renaissance/ECW
NAPO=1793-1815 Napoleonic
ACW=1861-1865 ACW
WW1=1914-1918 WW1
WW2W=1939-1945 WW2 World theatre
WW2A=1939-1945 WW2 European/Atlantic theatre
WW2P=1939-1945 WW2 Far East/Pacific theatre
MODE=1946+ Modern
SIFI=SciFi
UNAS=Unassigned, between some eras, none of the above, could be abstract
WRON=Wrong era entered by mapmaker

Schulz

I'am againt map rating idea due to so many unfair situations.

Maps that created eariler will have unfair advantages since they are already there for a long time and it will be easier for them to get high ratings whereas newer maps will even struggle to find someones to rate them.
WWI/WWII maps will have unfair advantages as well since their scenarios are based on history which were well known, multi-fronted interesting conflict. Creating a fictional scenario as exciting as these wars is harder than to create just another WWI/WWII map.
Giving every vote to equal weight is another problem. If there will be rating system, only old members should be able to vote because getting refined-fixed taste is not a quick process.
I don't even thing balance is the most important thing (unless its not extremely unbalanced) or graphics can be rated objectively. I personally give more importance to variation (having the most different ways to achive victory), speed and reversibilty than balance since the first ones cannot be fixed while the latter can already be re-adjusted by bidding.

I do believe there should have been way more section to rate games if this will be certainly happen.

Variation: How many different ways can be used to achieve victory? (More is objectively better)
Reversibility: Do players have good chance to recover their mistakes and turn the tide of war? (Games should be reversible enough to prevent earlier drop offs)
Luck: How much role does luck play? (Luck should not play major role)
Speed: Game timespan? (Long gameplay is drawback imho)
Balance: Is it at least reasonably balanced?
Elegance: Having not unneeed complexities, unit rosters, nations etc...
Historical Accuracy: Not meant to have perfect accuracy instead "does the game give feeling of the war its based on?" For example Japanese invasion of Vladivostok, India or Australia are probable outcomes although none of them happened in WWII but seeing Japan marching Moscow is a totally different thing.
Graphics: Including flags, colors, unit images, default zoom, relief tiles.

I am favour of to categorize maps based on their era. Total 5 categories: Pre WWI-WWI-WWII-Post WWII-Fantasy/Sci-Fi.

When any sub era category get sufficient amount of map, then they will be able to their own category. (Like, if there will be so many Cold War map then Cold War category will be created)

Also its possible to use both era and rate system like categorizing maps based on era than listing from top to bottom depends on its ratings.

TheDog

@Schulz
It would be great to have a detailed rating system, but it takes a lot of time and effort to evaluate each map, who will do that?
Some of what you are asking for is subjective and personal and cannot easily be graded/rated.

@RogerCooper does a very good review job with enough detail to intrigue me to download a map I would otherwise not bother with.
https://forums.triplea-game.org/topic/889/roger-s-scenario-thread
Thank you Roger!

Schulz

@TheDog That's why I am not favour of rating system. Categorizing maps based on era will do just fine.

Frostion

• A. how & where to manage the map category?
I think @TheDog lists a nice category system that covers most, if not all, possible maps types. Near identical map categorizations to this have been proposed several times before during past “category discussions”. And most old discussions also seemed to judge the current “High Quality” / “Good Quality” categorization as sub-optimal, since good/bad is a matter of subjective opinion, not a universal truth to be given by a moderator, developer or mapmaker.

I would think that Era/Theme categorization keeps getting proposed as it would clearly help the player navigate and explore available maps. Also, if the player already knows what he/she wants to download/play (maybe a specific map) he/she can go straight into the categories and find it in alphabetical order.

Ideally, if the above could be supplemented by a way to sort maps in the category by user rating (1-5 stars?) / number of downloads / number of registered times the map has been played etc. These sorts of ratings would make so much more sense than “High Quality” / “Good Quality”, or any other dictated rating.

As to not purge the map downloads of all in-development, unfinished, broken, forgotten maps, I think it would be wise to keep a category for these maps. Future potential mapmakers could then feel free to follow up on the development or find inspiration, tips and trick etc. Think it is obvious that more map-work end up unfinished then turn into finished maps, but that doesn’t mean that all the unfinished and finished work that needs to go into the dumpster. If a player (maybe wannabe mapmaker) downloads a map from a category maybe named “Unfinished Maps”, there are no expectations that can be broken in regards to quality.

• B. how can map makers be enabled to completely and accurately self manage categories?
Could the map not be “tagged” with some text in the map.yml or map.properties file? I mean if the sounds.properties file can have info like Sound.Default.Folder=generic, then a similar system could maybe give info on the category / categories? Am I right to also assume that tags should not only be used when downloading maps, but also when the player is locally choosing what map to play?

• C. how do new players when looking at the download list know which maps they should try first?
I gave my anser to this during A: Ideally, if the above could be supplemented by a way to sort maps in the category by user rating (1-5 stars?) / number of downloads / number of registered times the map has been played etc. These sorts of rating would make so much more sense than “High Quality” / “Good Quality”, or any other dictated rating.

@Schulz said in RFC: Map Rating to Replace Map Categories:

@TheDog That's why I am not favour of rating system. Categorizing maps based on era will do just fine.

I agree that a dictated rating / pre-given quality stamp is pretty problematic. It would be better to hold off any rating system until it is actually possible for players to rate based of playing experience.

TheDog

Pretend Im Joe Public, new to Triplea.
Joe wants a WW2 to play the Japanese in WW2.

If we have an Era/Theatre that narrows down his choice.

But as Joe Public is new to TripleA, he wants to be guided to a few maps to try, so 4-5 star maps? Maybe a 3 star if there are no 4-5 stars.

If we dont have a star rating how does Joe Public get what he's after?
He does not want to join the forum, so cant/will not ask a question.

Schulz

I think the yaml images are useful for this purpose, I also think the descriptions on the maps should be standardized.

They should be short and give most of the important informations.

For example this description is problematic:

"By LSSAH

Updated by Veqryn and Redrum

Preindustrial Napoleonic era European conquest

The empires of Europe are at war. The upstart French Republic has followed in the footsteps of their American compatriots across the sea and cast off the trappings of monarchy. Can this new light in Europe survive against a host of squabbling adversaries? Or will it fall victim to its own aspirations for an empire... Will the Russians be forced to sing La Marseillaise? Will the English see the Guillotine raised in Piccadilly? Or will the rule of the monarchs be ensured by the restoration of Louis XVIII to the Parisian throne? "

A new user would learn nothing about the game by reading it. These are good description examples:

"by Sieg Very large WW2 anachronistic map, based primarily on Revised rules, using mostly the New World Order units structure."
"by Frostion This map features 8 Axis and 12 Allied players. Play through WW2 as one or more Axis or Allied powers. Be free of restricting politics and non-aggression pacts. Purchase special units like SS, Colonial troops, Kamikaze plans etc. Keep track of Steel and Fuel income and availability on the map. Make use of the 25 different units this map has to offer. Enjoy custom map sounds and nation specific music. "

zlefin

I dislike the categorization proposed in the original post; but some other may work.

I'm wondering what would be most useful to users; and I find it's an inherently difficult question. To me, it seems that the users most in need of info on the map repository would be people who don't come to the lobby. For people who come to the lobby, they can ask people there, or simply download copies of whatever they see people playing; and they can learn about new games from others that way as well. But there are some users who never come to the lobby, who only play elsewhere with friends via pbem, or just play vs the ai. It's difficult to know what they need since one hears very little from them. If someone knows which map they want, they should be able to find it very easily for download; is there a list of simply "all maps? Or having that as a category, so those who know exactly what they want can just look up the name. I believe it might be useful to have an "ai playability" notation on maps, to account for some maps having features that the ai doesn't handle well or cannot handle at all.

It seems like one of the core questions on rankings/categories/assignments is whether to go User-based (where all the decisions are based on metrics gathered by users one way or another) or Curated, where community experts make notes as a group. User-based requires less maintenance, as that all gets done by the users themselves over time; while curated needs periodic revision and a process for determining who the experts are and how they decide disputes about ratings. But some things its quite hard to derive data from a user-base for.

It seems like having multiple categorization systems, so people can choose which one is most useful for them, would be good; albeit more work. I'd really like more info though about how different users use the map repository, and would be most useful to them. I wonder if we could make a survey or something and get people to answer it (especially people who don't use the lobby much or at all, though it'd probably take awhile to get info from such people).

One thing I note is that the WW2vX maps are amongst the most recognizable maps, and are played alot, but they often don't have nearly as good balance as some of the other maps nor graphics as good as some of the newer or fancier ones. Yet they clearly should in some sense have a 'high' rating due to their relative import.