Display Standard Deviation in Battle Simulator
-
Dear interested reader
Currently there is a pull request (PR #9490 [1]) adding to display the standard deviation of some battle attributes in the battle simulator. The attributes in question are the four attributes about remaining units and the TUV swing. The developers (which I am one of) currently discussing how to display this number to the user, i.e. you.
To quickly recap: The standard variance is a measure on how "wide spread" the individual results of the 200 (or whatever number you use) simulation rounds are. A small number means all results are very close together, a large standard deviation means the results of the simulated battles varied a lot. The more interested reader may look at the Wikipedia article [2] or other sources.
Currently, the idea is to display the standard deviation σ along with the average μ as
μ ± σ, i.e.Avg Attacker Units Left: 4.11 ± 2.58
See the picture in [3] for an (older) screenshot. Note that we will omit the total number "x / 10" of units as this would require adding parenthesis or similar to separate the plus/minus from the division, which would contract the long standing multiply before add rule. (Note: the total number is still displayed by the battle calculator elsewhere.)
The questions we'd like feedback to are:
- Would you (the humble user) understand what that number indicates and how to read it if displayed
μ ± σ? - More importantly, would it confuse or distract you or suggest to you a "better" or "worse" result than if only the average alone (i.e. status quo) would be displayed?
- What alternatives to display the standard deviation are there? (Maybe even with the total number) Remark: We have a single line of text to display that information.
- Instead of the standard deviation, should we show min or max of the sample? There was also the suggestion [4,5] to use two "asymmetric" values but this complicates the question how to display it.
Suggestions welcome!
[1] https://github.com/triplea-game/triplea/pull/9490
[2] https://en.wikipedia.org/wiki/Standard_deviation
[3] https://github.com/triplea-game/triplea/pull/9490#issuecomment-901708380
[4] https://github.com/triplea-game/triplea/pull/9490#issuecomment-902515786
[5] https://github.com/triplea-game/triplea/pull/9490#issuecomment-905790962 - Would you (the humble user) understand what that number indicates and how to read it if displayed
-
@emmanuel-perlow ± has my vote, dont we already show the ranges through the average if won numbers?
-
@ubernaut That format makes me think that is the maximum possible spread (which may be something of interest when playing Low Luck), not the average deviation from the mean.
As I've already stated elsewhere, I believe that knowing the standard deviation is almost pointless when the distribution is not gaussian nor close to it, so I think this feature is worthwhile only for possibly negative values (like the TUV swing). I think this feature is worthless for the winning/losing probabilities.
-
@cernel agree that existing displayed info should not be sacrificed if we are adding any new stats
-
An idea, I think if we add new numbers we will need some way to explain it to the user what those numbers are.
To this extent, I think what we need is an opt-in choice to show more statistics. To do this, I'd put a little 'gear' icon-button in a corner of the battle calculator window that would open a 'battle calculator preferences' window. We can start by first migrating the generic setting options from the battle calculator window to its new preferences window.
Then, if we have an opt-in checkbox, we could put some explanatory text about what 'advanced statistics' would be.
EG:
show advanced statistics | Shows 5th percentile and 95th percentile along with averages. Example: TUV Swing: 10 : 20 : 90, means that in 5% or fewer battles there was a TUV swing of 10 or less., 50% of battles had a TUV swing of 20 or less, and 90% of battles had a TUV swing of 90 or less.Implied though is a switch to start showing percentiles rather than averages. If our distribution is normal at all then the two should be about the same.
Stepping back a bit, I think part of the issue is that makes sense to add additional percentiles, but maybe not standard deviation. The reason against standard deviation is it still does not quite imply how 'tight' the results distribution is. Is plus or minus 5 a lot? How likely is it that we will have a result outside of the average and one standard deviation?
Bottom line, I wonder what if we switched to median instead of average. With median it makes a lot of sense to add additional percentiles as an additional opt-in option.
-
@emmanuel-perlow
I would read it that way that 4.11 ± 2.58 means that 4.11 is the most likely outcome and there is a range of 5.16 (2*2.58) in all outcomes, meaning there are results from 1.53 to 6.69.
But what happens, if the upper range is bigger than the lower (or vv.)? -
@wirkey @LaFayette @emmanuel-perlow maybe the best thing would be a probability distribution graph standard deviation could just be tickmarks and the min-max could be perfect roll scenarios
-
@wirkey said in Display Standard Deviation in Battle Simulator:
4.11 ± 2.58
Std deviation can be tricky. 4.11 would be the average outcome, the plus min 2.5 means most other results were clusted within that distance from the average. It could be the case that most results were clustered much more closer with a few very extreme results, or the clustering was pretty tightly grouped (no way to tell based on std deviation alone).
Hence, IMO we probably really want percentiles if we are going to add more statistics.
