TripleA development
-
@frigoref @Panther @RogerCooper @LaFayette
@magicstyck said in TripleA development:
I've been looking at the FeignException 404 on map listing. I know what is happening and why. I'm working on a test and coding a fix for it. I should have a PR in by the end of the weekend. I can help work on any other blockers after that.
When I posted this afternoon, I thought the issue was a broken endpoint in the spitfire-server, the code that runs the lobby. I have since confirmed locally, that is not the case. When I spin up the 2.6 lobby, I'm able to run a GET request on the /maps/listing endpoint without issue.
curl -i -H "Accept: application/json" -H "Content-Type: application/json" -H "Authorization: Bearer oijsad80lksjdflsdflk" -H "System-Id-Header: 1234567890" -H "Triplea-Version: 2.6" -v http://127.0.0.1:8080/maps/listing * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0) > GET /maps/listing HTTP/1.1 > Host: 127.0.0.1:8080 > User-Agent: curl/7.64.1 > Accept: application/json > Content-Type: application/json > Authorization: Bearer oijsad80lksjdflsdflk > System-Id-Header: 1234567890 > Triplea-Version: 2.6 > < HTTP/1.1 200 OK HTTP/1.1 200 OK < Date: Thu, 17 Feb 2022 05:39:42 GMT Date: Thu, 17 Feb 2022 05:39:42 GMT < Content-Type: application/json Content-Type: application/json < Vary: Accept-Encoding Vary: Accept-Encoding < Content-Length: 214 Content-Length: 214 < * Connection #0 to host 127.0.0.1 left intact [{"downloadUrl":"http://download-url","previewImageUrl":"http://preview-image","mapName":"test-map","lastCommitDateEpochMilli":1644988237583,"description":"map description","mapTags":[],"downloadSizeInBytes":1024}]* Closing connection 0I logged into the postgres database locally and deleted the record in the map_index table of the lobby_db (where data returned above comes from). This confirmed that if the table is empty, the /maps/listing endpoint does not return a 404. It returns a 200 and an empty JSON array.
curl -i -H "Accept: application/json" -H "Content-Type: application/json" -H "Authorization: Bearer oijsad80lksjdflsdflk" -H "System-Id-Header: 1234567890" -H "Triplea-Version: 2.6" -v http://127.0.0.1:8080/maps/listing * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0) > GET /maps/listing HTTP/1.1 > Host: 127.0.0.1:8080 > User-Agent: curl/7.64.1 > Accept: application/json > Content-Type: application/json > Authorization: Bearer oijsad80lksjdflsdflk > System-Id-Header: 1234567890 > Triplea-Version: 2.6 > < HTTP/1.1 200 OK HTTP/1.1 200 OK < Date: Thu, 17 Feb 2022 05:45:03 GMT Date: Thu, 17 Feb 2022 05:45:03 GMT < Content-Type: application/json Content-Type: application/json < Vary: Accept-Encoding Vary: Accept-Encoding < Content-Length: 2 Content-Length: 2 < * Connection #0 to host 127.0.0.1 left intact []* Closing connection 0According to closed PR that introduced the change (https://github.com/triplea-game/triplea/pull/9940), if I pass the Triplea-Version header then nginx should proxy me to the correct lobby. If the new nginx config is deployed, then it should read that header key and send me to the 2.6 lobby. If it isn't, then it will send me to the 2.5 lobby on every request. I sent the key with a value of 2.6 and was returned a 404.
curl -i -H "Accept: application/json" -H "Content-Type: application/json" -H "Authorization: Bearer oijsad80lksjdflsdflk" -H "System-Id-Header: 1234567890" -H "Triplea-Version: 2.6" -v https://prod2-lobby.triplea-game.org/maps/listing * Trying 173.255.225.250... * TCP_NODELAY set * Connected to prod2-lobby.triplea-game.org (173.255.225.250) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/ssl/cert.pem CApath: none * TLSv1.2 (OUT), TLS handshake, Client hello (1): * TLSv1.2 (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS handshake, Certificate (11): * TLSv1.2 (IN), TLS handshake, Server key exchange (12): * TLSv1.2 (IN), TLS handshake, Server finished (14): * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1): * TLSv1.2 (OUT), TLS handshake, Finished (20): * TLSv1.2 (IN), TLS change cipher, Change cipher spec (1): * TLSv1.2 (IN), TLS handshake, Finished (20): * SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384 * ALPN, server accepted to use h2 * Server certificate: * subject: CN=prod2-lobby.triplea-game.org * start date: Jan 30 23:55:24 2022 GMT * expire date: Apr 30 23:55:23 2022 GMT * subjectAltName: host "prod2-lobby.triplea-game.org" matched cert's "prod2-lobby.triplea-game.org" * issuer: C=US; O=Let's Encrypt; CN=R3 * SSL certificate verify ok. * Using HTTP2, server supports multi-use * Connection state changed (HTTP/2 confirmed) * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 * Using Stream ID: 1 (easy handle 0x7ff154810000) > GET /maps/listing HTTP/2 > Host: prod2-lobby.triplea-game.org > User-Agent: curl/7.64.1 > Accept: application/json > Content-Type: application/json > Authorization: Bearer oijsad80lksjdflsdflk > System-Id-Header: 1234567890 > Triplea-Version: 2.6 > * Connection state changed (MAX_CONCURRENT_STREAMS == 128)! < HTTP/2 404 HTTP/2 404 < server: nginx/1.18.0 (Ubuntu) server: nginx/1.18.0 (Ubuntu) < date: Thu, 17 Feb 2022 06:27:01 GMT date: Thu, 17 Feb 2022 06:27:01 GMT < content-type: application/json content-type: application/json < content-length: 43 content-length: 43 < * Connection #0 to host prod2-lobby.triplea-game.org left intact {"code":404,"message":"HTTP 404 Not Found"}* Closing connection 0So the issue is not a code problem but a deployment one. One of two things is incorrect in the production environment:
A: The nginx web server does not have the latest config deployed to it to properly proxy requests to the correct lobby.
B: The 2.6 lobby as a whole, has not been deployed or is setup but running a 2.5 lobby.
Until the 2.6 lobby is deployed and the nginx web servers config is updated, all 2.6 client requests for maps are going to return 404.
This card I found on the Project schedule (https://github.com/triplea-game/triplea/projects/25#card-75012745) suggests that the 2.6 lobby has not been deployed but can be at any time. If this gets deployed, we can close 6 or more issues and be closer to releasing 2.6.
-
@magicstyck Nice work
yea I'm not aware of a 2.6 lobby yet. @RoiEX do you know if one is up -
Before you invest too much time, please test the download maps - issue using the current prerelease.
I was just able to download maps using 2.6.571.I don't know what might have fixed the issue - nobody told us. So I was surprised when it just worked.
Don't always trust TripleA when it comes to rules questions. Know the rules before you start … and better check what TripleA has done.
-
@ubernaut
Feel free to change it. It is not my list, but ours and everyone is welcomed to keep it updated. -
@LaFayette Can you explain to the group what the download maps issue is and why it was fixed with 2.6.571 according to @panther?
(see also the analysis made by @MagicStyck below)
-
Don't always trust TripleA when it comes to rules questions. Know the rules before you start … and better check what TripleA has done.
-
@frigoref said in TripleA development:
@panther can you please also label the issues you consider 2.6 blocking issues?
Do you need help for this?Is there anyone interested in assisting panter?
Yes, I'll do. At least those I am able to identify, such as those named above.
-
@panther Thanks for the link! If I understand LaFayette correctly, the "find the server" problem should be solved, but the database might be crashing sometimes. I offered to help with an issue there if I get access to it and he is about to make this possible.
However, he also told me that he made some adjustments so the database shouldn't be crashing anymore.
@LaFayette Please get in contact with me to explain on the access part and leave a comment here on what the currently remainig issue (for the download maps topic) is that needs solving. -
2.6.571 and the current 2.6.574 both give the same 404 error. That is because the client still defaults to prod2-lobby.triplea-game.org (https://github.com/triplea-game/triplea/pull/9682). When you change the Engine Setting to point at prod.triplea-game.org you get the TLS certificate error discussed in that PR and you are still unable to download the maps.
Reading through the issue here: https://forums.triplea-game.org/topic/3109/unable-to-download-maps-in-2-6-535/13 there are still issues with the production environment for 2.6. They are as follows:
- No valid certificate for prod.triplea-game.org
- database is crashing for prod.triplea-game.org - includes but not limited to disk space issue
The client 2.6 client gives you the link to pull down the maps manually to get around the 2.6 lobby server not working.
-
Of course I have switched the settings to Pre Release in Engine-Preferences / Testing before, as advised by @LaFayette .
To me my "improved" experience is that even when changing to Pre-Release the download maps feature did NOT work until my new test yesterday.
Just for completeness.
-
When setting up my dev environment for this project over two years ago, it was a lot of trial and error to get it working.
Which OS are you using? If other than Linux with IntelliJ, then we have an active need for that configuration to be well documented. I am personally frustrated when any developer sets something up for themselves and does not document how to do so, leaving the rest of the team to independently re-discover that configuration.
So, I'm frustrated to hear that even then it was trial and error. There have been a significant number of improvements since then and it might be significantlly better..
TripleA has never had a very robust and up-to-date documentation system. Instead it was a variety of newly written documentation, some of high quality and of lesser quality. I can say now though that instead of having around 4 independently written different copies, they have all been dumped into a single location. Some attempts were made to prune this further down and organize. I'm not sure I like the 'how-to' vs 'reference' organization 100%, it's not flexible.. How to go from here is an open question. For 'set-up' documentation, particularly for OS's I do not own, I can only implore that contributors do the right thing and update documentation as they go along and follow it, and fill in any gaps. There has been work in this area and perhaps now it is far less trial and error.
or instance, the legacy documentation lays out a design that the current software begins to deviate from. How much still applies? What does not?
The legacy design starts to fall apart when details of multiplayer and bots were included into the code base. It's historical reference, has been for around a decade if not longer. The reference is maybe useful to know which principles the original authors were going for.
Is this a hoop per-say? Should we delete the legacy documentation outright? Archive it somewhere? Add an explanation of why we still have it at all?
Any change to the code proposed as an issue with how TripleA applies, even if it only affects the supported rulesets, is met with hostility based on "this might affect my game which doesn't use that ruleset", "might break game saves", and "shouldn't be considered because I don't agree with the Larry Harris accepted interpretation of the rule". There wasn't even a PR for the fix yet, just a conversation. https://github.com/triplea-game/triplea/issues/9067
This is a problem. There is nuance. The issue did help avoid investing time to do a PR and then have a long conversation. It does seem that discussing a change first avoided some pain.
re: "might break game saves"; this is not meant to be hostile. It needs to be accounted for. Right now we try to do save-game breaking changes rarely. It is incredibly impactful to players to have save games break. It is a design flaw in TripleA and is a major hindrance for development. If an update is going to break save games, it must be done at a time agreed when we will break save games. This makes a number of updates very difficult to do, if not impossible. Hence it is a high priority item we fix this problem. Hopefully the next time we break save games it will be so that we can move to a system that has a hope of genericly backward compatible save game changes.
re: "I don't agree with the Larry Harris accepted interpretation of the rule"; There are a lot of opinions in TripleA and it can be loud. It is sometimes an exercise in focus and trying not to be distracted and achieve concrete benefit. It sounds like we might benefit from a document that states how we make decisions. I think the problem in the example https://github.com/triplea-game/triplea/issues/9067 is the conversation got too lengthy and differing voices useful. There are times to say, "I see the points you're trying to make, I'm curious and would like to hear from others and if they share your same concerns or have other thoughts."
No credit for work done.
Heh, you're telling me.. On PR: https://github.com/triplea-game/triplea/pull/7906, it does note: "Co-authored-by: ...", which should show up in the git history. I'm not sure exactly what it takes for someone to show up in the contributors list. If the lack of attribution on github is due to 'squash-merges' instead of 'rebase-merges', the attribution aspect would be a compelling reason to change merge strategy. I'm amenable to any changes that would fix this.
No response or direction at times. Conversations just die.
Unfortunately yes. I can't give enough attention to everything that needs it and more items than I know about drop off, and sometimes for the worse when they shouldn't. I'm open and have been advocating that we re-align TripleA in any way possible such that a group (a team) could be responsible for anything and never just one individual. If that thing is giving guidance around the code base, then finding a way to democratize that knowledge would be excellent. For a number of reasons we've never had many developers, so it's generally only ever been one or two people that have a broad view on the codebase at one time, which creates for quite the bottleneck.
I think we can probably help this by really clarifying where developer conversations should occur and providing an 'escalation' mechanism for people to flag they really need clarification. This forum could be a decent place for that, we could try to do something to avoid the error reports from flooding the issues log, or maybe use github conversations. I think ideally we'd have it so many people could answer technical questions, though I don't think that would ever be a reality. I think a second best is making it more obvious when answer is still needed. Sometimes doing a
@DanVanAtta bump ^works well. I agree that should not be needed but if something needs me and drop, nagging is okay.. Generally I would prefer we resolve the underlying communication system/problem more thoroughly though so nagging would be a more a last-resort instead of first-resort.I would suggest not holding up PRs based on non-technical issues. For instance, this PR (https://github.com/triplea-game/triplea/pull/9963) was not merged for over 2 months because of a disagreement on adding a comment line which could have really been added later.
With respect, I'd like to point out that is not accurate. It took over a month for me to look at the PR at all. The overall delay was constrained by my availability, not by any disagreement. When I first looked at that PR, I left a comment that I hoped would be quickly addressed and avoid a very minor element of technical debt. When I got back to the PR weeks later, I saw their was presumably a misunderstanding and a disagreement based on a faulty premise. I then merged the PR. I did not insist the disagreement be hashed out further before merging.
What does it take to agree to be a maintainer? What is the process? What does it entail?
Good question. This should probably be spelled out. Generally it's ownership of the releases in TripleA and TripleA operations. A person can start by doing release-blocker bugfixes, finding release blockers (eg: validating releases), and doing code reviews. I think there should be more discussion on this and something more spelled out.
-
@MagicStyck Your investigations and conclusions are correct & on track. You are hitting on the biggest items that need to be resolved for 2.6 to be launched.
If someone downloads 2.6, by default it'll connect to 'prod.triplea-game.org'
Currently the servers and environment are not fully configured for that, perhaps straight up not even deployed, and hence the 404s.If someone downloads 2.6 and explicitly selects to use 'prerelease.triplea-game.org', the explicit selection is used and the 2.6 engine then bypasses the reverse proxy config that is otherwise in place and is hitting the prerelease server directly (hence avoiding 404s).
Though, the prerelease server had been suffering from disk space issues; this caused disk to fill up and the database then to crash and be unavailable for any requests to it thereafter (even if disk space were then freed). To fix this, the DB needed to be restarted and this was being done for some time.. I recently found, about a week ago, what looks like old map files that were taking up around a third of the disk. We have the bots download all maps, all maps are about 5GB or so, and there was an extra copy of this floating around. Without that extra copy we should have breathing room and the problem gone.
I'm hesitant to call that really fixed as I'm not sure if the extra copies would not re-appear. I was allowing for time to test that before digging in further.
Can you explain to the group what the download maps issue is and why it was fixed with 2.6.571 according to @panther?
The version change is a red herring; it should have been fixed for the config panther was using for previous 2.6 versions as well. It just so happened it was fixed after downloading a new version. If there really is a difference between versions downloading from prerelease explicitly, that would be very interesting.
Side-bar, is there anyway we can move this 2.6 launch related conversation to the 2.6 conversation thread? https://forums.triplea-game.org/topic/2999/2-6-release-getting-close-need-volunteers-to-help-beta-test-2-6
-
@MagicStyck , re: chats falling off.
My hope is that anyone opening an issue would 'own' that issue and try to drive it to closure. EG: every month, mention, "hey, this still needs X from
@Y" Invariably that does not happen, instead maintainers (eg: me), has to re-read from the start the entire issue to understand what is going on. This does not scale.We had 'stalebot' previously to outright close issues where this was not happening and where issues were just getting old. Stalebot was incredibly unpopular, and even the stalebot message of "hey, this is being closed, please re-open and add a comment of what needs to be done" was routinely ignored (and instead there was quiet and sometimes very loud griping about issues being closed by stalebot).
This is overall not a problem I can really readily solve. I'm open to ideas on how this could be better. I'm just reminded after seeing some issues that are still open and not moving anywhere, further compounding the problem that issues are getting "lost". Perhaps this is argument for us to bring back stalebot even but close issues after a mere 2 weeks if they are not labelled as bugs. I don't really have any better ideas, at this point the old issues I'd say are hurting us as well as causing ill-well with the crickets. I barely have time to even respond to this forum thread, let alone everything else..
-
@lafayette said in TripleA development:
Which OS are you using? If other than Linux with IntelliJ, then we have an active need for that configuration to be well documented. I am personally frustrated when any developer sets something up for themselves and does not document how to do so, leaving the rest of the team to independently re-discover that configuration.
I use a Mac and Linux for development. I have Windows 10 machine (primarily for gaming). I'll have a look at the current OS setup documentation in the repo and see what I can add given my environments.
The legacy design starts to fall apart when details of multiplayer and bots were included into the code base. It's historical reference, has been for around a decade if not longer. The reference is maybe useful to know which principles the original authors were going for.
Is this a hoop per-say? Should we delete the legacy documentation outright? Archive it somewhere? Add an explanation of why we still have it at all?
Please don't delete it. I found it very helpful in seeing what the original intention was. The gap is where it is now and where it is going. I'll take a crack at organizing the legacy docs so the knowledge doesn't get lost but it's considered the source of truth for development.
If the lack of attribution on github is due to 'squash-merges' instead of 'rebase-merges', the attribution aspect would be a compelling reason to change merge strategy. I'm amenable to any changes that would fix this.
That would do it. I removed the 'squash-merges' and 'rebase-merge' from all my professional teams specifically for this reason. The contributors section is taken from the master/main branch as feature branches are deleted after merge. It reads the contributions from the history of that branch. If your name is on the commit, you appear in the contributors section. You can browse that section and drill down into the exact commits a contributor has made. Squash and rebase rewrite the history of the commits to condense them down leaving only a single person on the condensed commit. Echos are left behind where you can git blame a file seeing that someone else made a commit at the same time as the person listed on the commit but you can't tell what they did as the hash associated with their commit is gone.
With respect, I'd like to point out that is not accurate. It took over a month for me to look at the PR at all. The overall delay was constrained by my availability, not by any disagreement. When I first looked at that PR, I left a comment that I hoped would be quickly addressed and avoid a very minor element of technical debt. When I got back to the PR weeks later, I saw their was presumably a misunderstanding and a disagreement based on a faulty premise. I then merged the PR. I did not insist the disagreement be hashed out further before merging.
Fair enough. I had the intention wrong. I'm sorry. Thank you for clarifying.
-
@LaFayette thanks for closing my PR as you mentioned. In that case, I was purposefully trying to sidestep all the things that we've been discussing here by making it totally "under the radar" - no javadocs, no comments, nothing- just actually making the code the way it should've been from the beginning (IMO) since that data structure is both a list (reading the list of points from the file) and a "lookup" structure (to find the point associated with the name). In other words, the burden falls on the original developer to comment why
LinkedHashMapwasn't used and original order was lost, since that would be the default assumption/expectation.With all love and respect to @Cernel (he always has good points/ideas/feedback/context, but despite that...) it was really frustrating to have my intentionally minor/secret/quick/minimally-invasive PR get mired down in long discussion. So I see both sides of these issues- ease of contributing vs. burden on maintainers. I think it's just that TripleA has become an enterprise-level codebase (there are some great and powerful features in there) but maintained by hobbyists and we all have "day jobs"...
For anyone who has invested time in it, thank you! it's been a great venue with keeping up with old friends I used to play A&A with in high school.
-
@lafayette said in TripleA development:
Though, the prerelease server had been suffering from disk space issues; this caused disk to fill up and the database then to crash and be unavailable for any requests to it thereafter (even if disk space were then freed). To fix this, the DB needed to be restarted and this was being done for some time.. I recently found, about a week ago, what looks like old map files that were taking up around a third of the disk. We have the bots download all maps, all maps are about 5GB or so, and there was an extra copy of this floating around. Without that extra copy we should have breathing room and the problem gone.
Oof. Double whammy. If you need a hand, I professionally live and breath this type of architecture and services. I updated my profile with my LinkedIn if you need referrences.
The version change is a red herring; it should have been fixed for the config panther was using for previous 2.6 versions as well. It just so happened it was fixed after downloading a new version. If there really is a difference between versions downloading from prerelease explicitly, that would be very interesting.
No difference in the 2.6 versions. I confirmed that this afternoon.
Side-bar, is there anyway we can move this 2.6 launch related conversation to the 2.6 conversation thread? https://forums.triplea-game.org/topic/2999/2-6-release-getting-close-need-volunteers-to-help-beta-test-2-6
Absolutely.
-
@magicstyck hahaha if we all joined up and started a software company together, it would be amazing
-
@djabwana said in TripleA development:
@magicstyck hahaha if we all joined up and started a software company together, it would be amazing
:beaming_face_with_smiling_eyes:
That made me laugh out loud :face_with_tears_of_joy:
-
What about issues where there's nothing more the person who opened it can do; and the only people who possibly can address the issue are very busy (eg you)? I mean, I tried to push the long-standing problem affecting the Global map unit transfer in lobby bots; but you're very busy, and poking you repeatedly doesn't change that you're busy, and it's not something that can be fixed by someone else since it appears to be about deployments.
I just don't see any good ways to handle it; sometimes priorities have to be made and certain issues will just get put on indefinite hiatus because the only people who can address them are busy.
-
I feel like we're all mostly at least on the same page. Some problems certainly remain, but our perspectives are more aligned now - that is valuable.
I think @MagicStyck & @frigoref should gain system access to at least the pre-prod environments if not also prod so that they can help there.
I do strongly prefer having squash merges for a better history, and certainly at least rebase merges so that the history is linear. We have had several bugs where we were only able to track the problem down after bisecting the history. If the history were non-linear, we would no longer be able to do that. Some problems are amazingly subtle, I'm sure we would have had to roll back a years worth of commit if the history were non-linear. Perhaps the answer is to more readily give write access to those who want to contribute more than a single PR. This way those that are doing the bulk of a PR can be the one doing the merge and get attribution. As a complement and/or alternative - perhaps in the release notes we could explicitly mention the contributors since they may not appear in the git history properly.
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login