Is this the official place to report bugs?
-
Good feedback, we've had a number of historical problems to solve, all of these predating this lobby:
- users have had trouble finding where to report bugs at all, let alone knowing about forums
- admin attention being spread too thin with too much overhead to make much progress beyond just keeping up
- issues getting lost/scattered, this is two fold; first the official SVN bug tracker did not have good visibility and its queue grew too long, secondly bugs were tracked outside of the bug tracker which lead to reports getting lost (specifically these are forum reports, if a flurry of reports come in, it's hard to make sure they all get addressed, and often some fall through the cracks particularly when a fix takes a few weeks)
The second item I've viewed as particularly important, we have few devs/admins in this project and the overhead cost for them historically has been really high (arguably the full time job of 1 to 3 people). That's honestly why we moved to github in one way, the efficiency boost for dev helped free us up to do more; maps were moved so that they could be largely self service as well.
Thinking about our goal and pipeline, we want to get bugs fixed and for each one we have to go through a discovery, reporting, (dev) reproduce and fix and then push the fix to prod and then users pick up the updated version. The two bottlenecks there are with reporting, any friction that prevents users from discovering and reporting bugs is something we want fixed, and also fixing bugs is a bottleneck.
To fix a process bottleneck, we can add labor, add automation, remove tasks. To that extent, TripleA is fortunate to have a large and knowledgeable volunteer base, we can remove burden from devs and add labor if we are able to effectively to get volunteers to help with incoming bugs.
At the same time, we don't want volunteers to be bottlenecks themselves, so we need to think of the bug intake process as checking in to the airport, the volunteers are not the people doing the check-ins, they are the people walking around making sure everyone is okay using the automated check-in machines. As part of that, there is key information needed when reporting bugs so devs know what to look for and fix. Part of the inefficiency of fixing bugs is repeated asking for information that is needed before starting, to help solve that questions/check lists and automated tools are good answers so it's much easier more natural.
So, 2~3 years ago it was really not clear where to report bugs, and we did not have this forum. There was a concerted effort to try and improve efficiency and increase the rate bug reports by making the bug reporting steps relatively clear and directing people to the bug system. Hence, my concern when I realize we have developed a parallel flow here for bugs, which in an unstructured process really backtracks and could re-invite many of the important problems that we have almost left behind us.
Sorry for not having a lot of time to be able to shorten that down, it's hard to answer in short.. Hopefully the problems and motivations make it more clear we are just looking at a different set of problems and the shared perspective I hope helps.
With a really long story short, I thnk the way this is helped is simply by merging/combining the bug report section with the help section forums. Looking at the types and content of threads between the two, they are really similar. Once they are clarified and combined, and it made a bit more clear that is the place for questions and official bugs are put in to a bug tracker, I think things would be more clear for the new players who will be joining us over the next few years (an easy bug report process is important for them, and I'm most concerned about attracting and retaining new players, growing community IMO is top importance)
-
I believe the Bug Reports subsection was made because users would mostly not go reporting in GitHub, nor they would post in a specific place, nor the mods would keep traking the bug reports down and moving them to GitHub or even in a single place in the forum, but there were tons of bug reports lost spuriously in the various subsections, and diluted in between of all non-bugs posts in there.
If consolidating, I believe map bugs should better stay in "Maps & Mods", not in "Help & Questions", and preferably rather in a subsection of it.
-
@cernel said in Is this the official place to report bugs?:
If consolidating, I believe map bugs should better stay in "Maps & Mods", not in "Help & Questions", and preferably rather in a subsection of it.
Agree.
Thanks for background on why the 'bugs' category is needed - makes sense.
Is there agreement it looks to make sense to combine "Help & Questions" + "Bugs"? (please check out the thread posts between the two, they seem pretty similar topics). If that sounds right, here's a first list of potential titles for a new combined category I can suggest:
- "Help, Bugs & Questions"
- "Help, Bug Reports & Questions"
- "Help, Problems & Questions"
- "Bug Reports & Questions"
- "Game Help & Bugs"
-
@lafayette fwiw "Game Help & Bugs" jumped out the most for me.
-
I think it is a mistake to once again try to rearrange forum categories or remove entire categories in an effort to control how people report bugs and give feedback. The forum works fine. I can’t see the problems described in this thread as being related to bad categorization at this forum. People post about bugs here, and if they possess the know-how, they will post at GuitHub also or instead. If they don’t possess the know-how, somebody else will maybe post for them at GitHub if the bug they point out is actually a bug and/or if it is important enough for developers to tend to.
Some of the largest and hottest games in the world 2018 have both support and bug categories in their forums, to ask for help and to discuss (potential) bugs:
League of Legends has “Ask the Community” and “Report A Bug” as separate forum categories. (https://boards.na.leagueoflegends.com/en/)
Fortnite has “Technical support” and “Bug Reports” as separate forum categories. (https://www.epicgames.com/fortnite/forums)
PlayerUnknown's Battlegrounds have three categories: “General help”, “Troubleshooting” and “Bug Reports” (https://forums.playbattlegrounds.com/)
Warcraft has “Technical Support” and “Bug Report” as different categories (https://us.battle.net/forums/en/wow/)
And the list could go on … naturally … as it is a good idea to have both help and bug categories at a game forum, and serious games therefore have both. Meaning open doors for players to both discuss and report game related problems.
“Once they are clarified and combined, and it made a bit more clear that is the place for questions and official bugs are put in to a bug tracker, I think things would be more clear for the new players”
I think it is wishful thinking to believe that cutting down categories and mixing bug reporting with player support and inquiries (“Help & Questions” and “Bug Reports”) will do anything good or make bug reports more clear. On the contrary I think that it will make it even harder for devs to identify bug reports (as they will be mixed with a lot of other inquiries) and it would also be harder for players to find out where and what to post.
I am sorry if I am repeating myself, but I sense that we are maybe trying to fix something that isn’t broken, maybe even creating more problems in the process.
-
@frostion said in Is this the official place to report bugs?:
. People post about bugs here, and if they possess the know-how, they will post at GuitHub also or instead.
This is the thing, not having that know-how means we have failed to provide a reasonable way to report bugs. Having multiple ways to do that makes the problem worse. To boot, there is confusion and if you get help on how to post bugs, depending on who you ask, you'll get conflicted answers. Bugs on forums, if they never make it to bug tracker, will get lost, and if they do not get lost then they add a really excessive amount of overhead. Scenario, dev finds a bug report a week after the fact, asks for some important information and original poster is long gone. Then to boot, that thread goes to the top and there is no way to mark it's just waiting, so others devs are going to open that, read through it and lose time realizing it's in a waiting state.
@Frostion , that list of games, none are open source. The bug trackers for each of those would be closed source products not available to the general public. So in each case it seems they all have one way to report bugs. Surely those are not the official bug queues developers use. There is a reason bug tracker is a dedicated piece of software, whether that is github issues, JIRA, svn bugs, or what not - there are some key things a bug tracker needs to do, and forums does not satisfy those needs.
In the examples, I'm pretty sure the bug forums makes a lot of sense as it's the way for users to report bugs, it's split and different enough from other categories that the posts there share a common theme.
For us though, I'm seeing almost the same thread topics between Help and Bugs. I don't think our Help section is currently useful because of that, it's a toss-up to post in help or bugs. What makes it bad, is that we don't have an official policy for how we accept bugs anymore. Again, a bug report in forum that does not make it into tracker is a loss for everyone; so whatever we decide we need to be able to avoid that. Again, community volunteers should not be the ones taking bug reports, but they should be helping to make sure they are reported properly.
On the contrary I think that it will make it even harder for devs to identify bug reports (as they will be mixed with a lot of other inquiries) and it would also be harder for players to find out where and what to post.
I have trouble seeing the evidence for that. There are many wrongly placed items in the help:
- slow AI turns after 5+rounds https://forums.triplea-game.org/topic/939/slow-ai-turns-after-5-rounds (this is a performance bug)
- java error - https://forums.triplea-game.org/topic/926/java-error, this is a straight bug, should not be in help
- https://forums.triplea-game.org/topic/890/show-code-for-dice-algorithm, this is a feature request, should not be in help
- https://forums.triplea-game.org/topic/880/error-message, another bug report
- registration email never received: https://forums.triplea-game.org/topic/862/registering-email-issue, bug
The above list is from the first two pages, I think that already shows it's hard to identify the bug reports that are posted, so I do not agree with " On the contrary I think that it will make it even harder for devs to identify bug reports", as is I see it is already 'harder' for devs. Devs essentially have to scan forums, all threads periodically to check and make sure that issues are not getting lost here (which happened in old forums, I see it happening here again too, as some of those bug reports in help are new to me and there are fixes that could be done to help those and are not done yet).
-
I think there is one point that is getting in the way, whether bugs need to get into the bug tracker or if it's sufficient to just list them on forums. As far as that goes, this project always has had a bug tracker and always needed bugs to get into the tracker.
The old bug tracker I think we collectively keep forgetting about, but it's always been around and still is even: https://sourceforge.net/p/triplea/bugs/?source=navbar
As far as it goes whether devs should use or need bug tracking software, it's a debate I never thought I would have necessarily. As is, forums is not set up to be a bug tracker, and that does not work for development.
There are lots of online resources about software project management and how a bug tracker fits into a software project (a bug tracker is one of a few central project tracking components, up there with source code management software):
-
(some) people will always post about bugs in the wrong place, even if a category spelled out "NOT FOR BUG REPORTING!" I think we could see bug reports being posted. One of the forums listed above has a similar capital letter announcement in a category description, so I think what we see at our forum is not a unique phenomena. I think this is a good opportunity to remind members with moderator privileges to move all wrongly placed posts they see, of course only if the post can be determined as wrongly placed. I would say that many posts are actually a bit hard to categorise or define as bug reports, so no wonder bug reporters some times pick the wrong category.
I would say that the only way to have all players know exactly how and where to place bug reports would be to have a bug report button or bug report info at the TripleA program's main menu. The simple version could just be a pop up with text instructions and a link. The advanced maybe a fill out form and maybe a possibility to upload a file (screenshot or txt?). Just some sort of ingame bug reporting help, then devs could just stick to these reports without worrying about bugs being lost or misplaced. Maybe the bug tracker chosen for TripleA should be very user friendly? If the bug tracker is to complicated to use for the common player, I fear we can expect players to not report the bugs they encounter.
There must be some old experiences with the old bug tracker. Was it only the same few "power users" who posted about bugs? Or was the tracker friendly enough for everyone to use? I think that the format chosen will determine if the devs want everyone to be able to post, or if they feel fine with just the elite few players and their reports, just like Github feels now. Nothing wrong in my mind with the latter if the development runs fine without input from the masses, but it would maybe be a bit excluding. And in this case, the forum could just remove all involvement in regards to bugs, and everyone can just point towards the official tracker.
-
I mostly agree with @frostion, but here it seems to me that we are going in circles a lot, also repeating stuff that has already been said several times in the many pages long Bunker topic I linked, and the more a topic keeps going in circles, the more the text added to it, the less the chance that all the topic will be fully read, thus the more and more the circling; this one might have already gone out of control, practically.
@frostion said in Is this the official place to report bugs?:
There must be some old experiences with the old bug tracker. Was it only the same few "power users" who posted about bugs? Or was the tracker friendly enough for everyone to use?
I've almost never seen known active people in the lobby posting in the old tracker, but really the matter is of little relevance, as for many years in sourceforge TripleA has remained very stable, so the addition of new bugs was near to non-existent, and veqryn pretty much knew all that was pending, that were all marginal or arguable matters. Really totally a different scenario from today, or the last years anyways.
-
For several years when TripleA was really in icebox almost all new I was seeing in the sourceforge tracker was veqryn closing as invalid or duplicate rare bug reports from people whose names I had never seen before.
-
Thank you @Frostion , @Cernel for the dialog. We are talking in circles a bit, but I do think we are at least talking from a common point of view now. TripleA is pretty complex and it's interesting to think it is a half dozen people running this with a larger community.
The thing that I have a hang up over, just a problem I can't see how to solve, if we ask for bug reports as a bug report in forum, it will always fall on someone to move it to the official tracker. Maybe that is good, and if someone then sees that happen a few times and they themselves become a more active volunteer, then they'll cut bug tickets directly into the tracker. Really, it's how do we want this designed?
I like the mention that the old tracker and new are pretty different. That is a good important thing to keep in mind and the fact it's easier to see/add/update tickets in github issues is significant. The old tracker, I would have supported that being for absolutely just the devs, I still can't figure out how to mark tickets as fixed in it
I think we have two decisions to make:
-
Do we encourage users to go directly to bug tracker, or forums? There are in-game links, help menu, lobby message that direct users to report bugs directly to bug tracker. We could update those, but overall it's a process/flow question. I think we have two realistic answers here:
a) let's encourage bug reports to go directly to bug tracker
b) encourage bug reports to go to forums, encourage experienced users to user tracker (github issues) directly -
Do we merge 'help' + 'bug' categories? If we go with 1a, then it probably makes sense to do this. If we go with 1b, it will make complete sense do keep the bug category.
If we can formalize a way that 1b works, that would be best from a labor perspective. People that are experienced and hanging around the forums can add their labor which saves from dev efforts (we want devs coding). At the same time, repeating myself here, we do not want the forum admins/moderator volunteers to be a bottleneck either, which makes me think this might be a risky idea. That has me leaning towards 1a, and then combining the help and bug report categories, which is effectively a "we'll help you here if you're not sure, but the official bug reports go elsewhere"
Sidenote: i'm heading out of town for a couple weeks and will not be able to follow up. It is just as well IMO, I think i've exhausted my thinking about this and have hit a brickwall. It's hard to balance getting new users onboarded, bug reports going where we want, using available talent and labor, and it being clear and simple.
I do really appreciate we are working through what has been a contentious issue : ) I think we'll get this solved for real. While migrating there was a lot of updates for old bug tracker to new, that was a dev only effort, and we had a big split before in the different groups within TripleA. We're now a more united community IMO, so it seems appropriate that we're now dealing with, okay, this was how 'devs' dealt with bugs, how should TripleA as a full holistic community deal with them?
-
-
I think option 1b (Encourage common players to report bugs at forum, and advanced users to go directly to Github)
This is because I don't see a bottle neck. I see the forum as a filter where possible bugs are posted, evaluated by more competent people and identified as being an actual real bug or not. Also, at the forum, it can be determined if the reporter himself is capable of and competent enough to post at Github, describing and discussing his bug. If not, someone else could/should go to Github with the bug ... in best case the reporter has the drive and motivation to learn to do it himself.
I think this option ensures the most bug reporting (and broadest community inclusion ... something needed to recruit and create development and mapmaking interest
) and also it should ensure some level of quality when it comes to what is posted at Github, so Github does not become flooded with questionable and debatable reports. Hence the "filter".I must admit that I am not at all competent to say what would or could lead to the best working environment for developers at Github, that is of course something the devs should know and determin. So I wish more devs would give some input here. But I do feel that the rest of my analysis is fairly reasonable
-
1b works if it is formalized. The 'bottleneck' comment should not be misconstrued please, it's more a matter of looking at
the process flow. One process being considered (1a) has a direct intake to bug tracker, while 1b has an intake to forum and output to bug tracker. The conversion from forum post to bug tracking ticket will not be immediate nor 1:1, by definition a bottleneck as we will be waiting for humans to do something (and no matter how diligant or fast, that will introduce some delay into the process). Whether it is a big impact or not is another and important matter. The impact will be inconsistent, we need to design this process so that this will not become a problem and we will not see a process break down.How does 1b break down? what do we need to avoid/solve?
-
appropriate culture of 'always do something' not instilled. For example, in: https://forums.triplea-game.org/topic/920/some-help-me-please/3, the "your internet is not avaialble" warning message was misconstrued and reported as an error itself. A response of 'ya, that is normal, just close it and continue' is an example of where we do not have that culture. No UI notification should ever be routinely ignored, and secondly the warning message is clearly not helpful in its current form. So the action item is to either remove it or fix it, but we can't just say "yeah, that was your problem", because it was not the users problem, it was our poor design and lack of time/labor for code development (which is compounded by really high overhead processes and high overhead code).
-
potentially times of low capacity, people get busy; If we have only 3-5 people moving tickets, and 2 or 3 of them happen to coincide by being away and/or busy at work or what not, then we'll have pretty limited capacity and forum posts would build up. Dev's would jump in for these situations, but that is then reducing time available for devs to code, and it's hard to then justify having the extra reporting step.
-
we simply lose track if something has been moved and do not get around to it. This report: https://forums.triplea-game.org/topic/957/pbem-pbf-combat-move-check-box/8 is potentialy an example, the reported bug has not made it into tracker. With more time, the post will fall from the top and would be unlikely to ever be fixed.
For this process, the success and failure metrics I would choose would be (so these are numbers that we should work to optimize and track to know that we are doing well):
- number of forum posts that are handled without needing a ticket. Sometimes questions just need to be asked. (We would need to be sure to count this metric very conservatively, most user reports should result in some action, even if the 'system is working as intended', if a user can't figure out what that intent is, it may as well not be working).
- number of forum posts that result in a bug tracking ticket or are de-duped against an already open ticket. If the ratio of this number to the total number of posts start to drop, then we are falling behind
- the amount of time between when a forum post is opened and when a bug report is created
- number of bug reports (forum or github issue, any kind of report) that is filed in the wrong place
- number of github bug tracking tickets missing important information
-
-
Also, in past there used to be in lobby a link to where to post bugs. I guess you removed since you can click on Help (not so obvious, in my opinion), but just saying in case.
Maybe off topic but still talking about bugs, this is what happens when you are in the lobby and want to see how to report a bug:
You click on "Help"
Click on "Help Page"
Get redirected to this page:
https://triplea-game.org/help/
making you click to go to:
https://forums.triplea-game.org/category/10/help-questions
at this point, someone that is searching for bug reporting will hopefully click on:
https://forums.triplea-game.org/topic/526/how-to-submit-a-bug-report
where you find the link to opening a new issue in the tracker (need to register to GitHub, of course).
Side note, at this topic, the:
"Open Bug List"
and
"Lobby is on Fire"
links both have GitHub telling you:
"No results matched your search."
On the other hand, if the user would, instead, decide to click on:
https://forums.triplea-game.org/topic/13/faq-and-help
and then on:
To submit bug reports: http://www.triplea-game.org/bug_report/
he would get a:
404
File not found
(or at least this is what I'm getting)So maybe in the moment you get a bug while you are in the lobby, knowing how to report it may be not very clear. Maybe it was better the old practice to have just the link to where to bug report at the start of lobby chatlog (your decision), for the users to copy-paste.
Anyways, not so important, as I believe lobby users very rarely report bugs anywhere (sadly), so having a link there won't do much to increase bug reporting, I guess. -
@cernel Thanks for tracing that through, hard not to chuckle at the absurdity.
The bug report link had the virtue it was simple. The above flow is broken, I don't think we solve that by creating a 'bug reports' section on forum though as it implies this is the place for bug reports and that becomes confusing. A formalized process of moving bug reports from one place to another I do not think will be sustained over the long-term (in the short it does not seem to be working, I note a number of bugs that are being 100% worked on here, and did not get proper tracking visibility).
If we really want to unify logins we can have forums to use github API for login, then that could be the one login for every TripleA system. There are some real benefits to that as we have less custom code to maintain and that would reduce our engineering costs.
So I think the bottom line is having a 'bug reports' forum section here is well-intended, but fundamentally confusing as this is not our flow/process for reporting and tracking bugs. Merging this category with 'help' section and fixing the bug reporting flow so it takes a person to the right place the first time would be good next steps.
-
Why those next steps?
Well, how should this get fixed? We need something, concretely how to instruct the computer what to do.
If we update the help menu to have a forum link for reporting bugs, then that will be where bugs go. Sadly, this is not bug tracking software and issues are going to get lost/missed (recall when map updates were driven by forums, I counted that a good percentage were just dropped and never actioned, that was largely in part that forums track active conversations, they are not designed for priority and task tracking).
So the answer seems to be to have the help links be the single "submit a new bug report" link that we once had. I'm wondering how to work in "see the forums if you are unsure of what to do and if you are not sure this is a bug", but that confuses everything and gets back to the problem that eventaully we will not move everything to bug tracking and that will cause items to be lost (and if lost, then reporting here is just not going to be as successful as it could be).
This is for sure an open conversation, I can be pursuaded, but my current thinking is:
- it will not be successful to do task tracking in forums, we need a bug tracker for it.
- we can route tickets through forum if we have a real and sustainable and formal process to move those tickets into bug tracking
- over the long run, even short term, getting such a formalized definition and sustaining it for the life of the project (hopefully a few more decades at least) will eventually fail
- splitting the instructions to route things to forums makes those instructions more difficult to follow and maintain, it adds to the confusion and the 'trust no documentation' because we have 5 copies of it all and all are slightly out of date
I think that leaves trying to make our bug tracking queue very obvious and the one place where we do that kind of reporting /tracking. Just not necessarily seeing how we can solve it otherwise without running into additional problems (and thinking historically to how tracking failed with the SVN bug tracking, and noting how things ran when we had no active forum and only github issues; things work better when there is process clarity and transparency)
-
Good news and a bit of a conversation pivot. How bug reports make it into the tracker will soon be a problem for just admin/dev.
We're introducing an http server to the mix and that let's us build a few things pretty quickly. The first feature to make use of this is a 'error report' feature for the client that is in progress.: https://github.com/triplea-game/triplea/projects/4
The original idea was to upload stack traces to the server, with a little bit more we can add a text field on to it to get a free-form input so we can submit bug reports directly from in the game. Hence, it'll be up to dev/admin how the reports are processed after they are entered, they will in a DB and we could potentially do things like automatically create bug tracker tickets when new items show up.
A logical add-on to this would be to add the capability to capture and upload a save-game while uploading the error-report. The http-server/client components are looking far enough along that we can potentially start thinking about how that process would look.
