QA Team + TripleA Release Process
-
Background: TripleA QA team has been operating for some time:
https://forums.triplea-game.org/topic/268/volunteers-needed-early-release-testing-groupWe have coordinated mostly via gitter channel: https://gitter.im/triplea-game/QA
We coordinate testing via a pretty simple google spreadsheet:
https://docs.google.com/spreadsheets/d/1CSZ4s7wBLcvgue_IHwl-CbjQd95qj8hhR8m89ngCzbU/edit#gid=0When there are multiple green bars lining up, meaning multiple people have reasonably checked as much of the game as they can, that version of the game can become the latest release.
-
Why the QA team?
The QA team is essentially a way to coordinate multiple people to test the software before it is released. The team was created in response to a number of problems.
- Development team had trouble tracking results and general confusion on "is it safe to release?". With a defined policy and tracking, we can now look at the testing results and know when a specific version has been confirmed to be okay for release.
- the TripleA code base is very difficult to modify and repeated releases kept introducing small but extremely significant bugs. Regression testing for any of these is a huge effort, and needs to be done repeatedly as the code base is constantly modified. It's too large of a task for just one or two developers, hence the QA team.
How will we know if the QA team is successful?
- The number of major bugs reported by users to github issues will go down and head towards zero
- The number of times a release is 'rolled back' will go down and head to zero
Where is the QA team heading?
- Still formalizing how we operate, but we're getting pretty close to a stable operating process
- Long term try to work with development team to reduce the number of items being hand tested
- Short/medium term, formalize testing a bit, a few things are getting through, perhaps a testing check list of all features and cases that need to be checked would be a good start
-
Want to update this as it's been some time; The QA tracking spreadsheet seems dated at the moment because we've had an incompatible release for so long. When we are building compatible releases we can release minor versions pretty frequently. There needs to be a discussion on the release schedule and if there should be one.
Beyond all that, the best way to help and be part of the QA team is simply to download a latest prerelease and find bugs in it. File bugs here: https://github.com/triplea-game/triplea/issues/new
-
@LaFayette Though I am constantly testing pre-releases, I have not updated the spreadsheet for some weeks now. The reason for this is the sometimes high frequency of updating pre-releases per day. I see me updating my pre-release version sometimes maybe three times a day to continue using the latest version and testing the latest pre-release.
Sometimes the changes that lead to a new pre-release are not comprehensible to non-developers.
Does that change affect gameplay? Is it "just" something in the background? Do I have to restart testing because PR xyz has been merged? Can I just move on just using the new pre-release?That is why I hestitate to permanently update the version that I am testing in the spreadsheet.
-
Perhaps when we get closer to a release (as we happen to be now), the spreadsheet will be useful for coordination to hone in on a stable version
