RFC - In-Game Map Uploads
-
Github repository is probably not the tool for the job, it might have a small learning curve, but I am still not persuaded to use it, it requires another tool to upload a game, a quick search of the games I have and the smallest is 450 files.
Github is just an obstacle that I dont want to over come as it offers nothing that I want to use, however the restore points for an xml might of tempted me, but no, I will stick with my many backups and comments in my xml files.
.As mentioned I would like to just upload 3 files, as 400+ files takes too long.
But the process is
zip, upload
change, zip, upload
But, there is is no download, extract as only one person will be doing the changes.
.As for collaboration I would ask outside of Github for help, email back and forth, share on Google Drive , so no need for Github. Game maps tend to be by one person, that's another reason why Github does not suit one person as its really designed for collaboration.
.
.
ps. the Fallen Empire has 416 base and 416 relief tiles, yes its a big map, it was design by alkexr, but never finished, the rest of the files I need to tidy up, but it runs well as a single player. -
Github repository is probably not the tool for the job, it might have a small learning curve, but I am still not persuaded to use it
I'm not trying to pursuade you to use it. If you want your maps uploaded, it's what you have had to do. Maps have always been stored in source control (previously sourceforge). The biggest difference is the work is now self-service (relatively easy) and development does not manage it so closely. This frees everyone up.
-
If the 100 file limit is a bottleneck then you can use a desktop GUI that makes it easy to do the needed git commands locally and then push all files at once.
-
Github enables collaboration, but I use it personally for solo projects as well and am aware of many other examples. The email'ing, sharing via google drive, it all could be removed by commiting to the repository instead.
-
I think the solo led collaboration is common but it's not always the case. The really popular maps have many people working on them, particularly over the years (think long term). For example we've had maps where the unit images were upgraded.
-
There are more benefits to having github repositories, for example all XML is available at a web link, makes it really easy to see how other maps were built and examine them without downloading and exracting. It also makes it easier for admins to go in and correct issues, and if used would allow map makers to link to their XML and have others look at it easily.
-
-
I'm starting to lean towards the thought that storing map zips in database only would be a mistake. The repositories provide a lot of value from different perspectives:
- all files available at a web link.
- an automatic history of changes available at web links, this is critical for an editorial review of maps to verify what has changed
- automatic packaging of files
- capability to do bulk updates.
- capability to do bulk searches. Having all maps checked out locally, it's some easy commands to search all XMLs in existence.
- capability to do bulk file adds.
Diving into some of those details, we have built an in-code test that downloads nearly every XML that exists and verifies that the map parser is able to read them without any errors. If we mess up the map parser somehow in an obscure way we've a good chance of finding that. Every time we change code, we run this test automatically as a requirement for that change to be accepted.
The file history is automatic and really critical for editorial review. If we were to have zip files only, then the already nearly non-existent map-admin team would then be slammed with having to open every new zip file uploaded and fully verify the entire map rather than having a web link tell them exactly what changed.
With all maps checked out, it's possible and has been done to bulk change every XML that we have to apply corrections. Similarly being able to search all files has let us find features which are not used, or find features that are heavily used or lightly used, get statistics on which images are duplicated the most and least, and generally it's provided useful insights that would not be possible if we had to download from scatch every zip and extract.
File adds is where repository does shine. Consider adding a standard config file to every map. That is a very likely scenario coming up as we break up and distribute the index 'triplea_maps.yaml' file to each map (which then avoids needing to have the single centralized index file). So for example, we add this config file to every map: If every map is in database and uploaded as a zip, we need to coordinate with every single map maker such that they add the file or download the latest zip and merge. If we fail and a map maker uploads a zip without the config file then they'll inadvertently remove it. Checking every map consistently keeps the config file will become non-trivial and painful and a source for troubleshooting and error. Overall, a messy scenario that incurs more work than we are doing today.
Getting to the point, github repositories provides: file storage, version tracking, access control, web hosting & automatic packaging. That's quite a lot to be getting for free and would be an enormous amount to build ourselves from scratch. Perhaps making bulk uploads easier with good step-by-step instructions is a lot of what we are missing.
-
@LaFayette You need to verify for missing images. My experience is that is the number one problem with maps, including 2 in the current repository.
-
I had easier time learning to create a map from scratch than understanding Github sufficienly in spite of Frostion's guidance. After that it is easy to manage and of course very valuable if map user can achieve to do it.
Would Bitbucket be better alternative?
I think nobody should be able to in charge of any project other than the original creator besides fixing bugs.
-
@RogerCooper I found another non-working map in repository, "Caravan Plain".
-
@RogerCooper A map validator is a highly requested feature. I've also been thinking it might make sense to automatically do hue shifting from existing images to fill in missing images rather than the hue shift be controlled by configuration (just make it automatic).
-
@Schulz What exactly did you find difficult in Github? To me the web UI is as easy as it could almost be. It is almost literally click a button, drag and drop files, then click another button to submit the changes. Hence, I'm very interested to know where the complexities are.
No, BitBucket would not be a better alternative and it has none of the web UI features, you would have to 100% use a Git client that you install on your computer and then BB has other issues. We are not migrating hosting system.
-
@RogerCooper Do you have a link? Maps are what I am using to try to learn TripleA some more.
-
I think this thread is really getting off-focus. The question is if we want to invest 3 months of TripleA development time to replace how uploads are done and bring it in-game. I think intrinsically it is a good feature to have in-game, but the cost is significant and particularly for something where there is a very decent UI someone else has built that we can use free of the maintenance overhead that would come from a custom creation.
Preventing broken maps from staying in non-experimental categories is a problem no matter where those maps are hosted or how they are uploaded. Ensuring maps are followed-up on, that map makers don't half-complete maps is another similar problem that is neither made worse nor solved based on how maps are uploaded (though, if we did get more uploads, it would put pressure on that problem, but the solutions to that are independent of how we do map uploads).
Re-iterating - Main Questions so Far
-
Is Github repository upload actually that complicated, or is it that there is a small learning curve that is hard to approach?
-
What is difficult about repository upload exactly? Where do we need to streamline instructions if we were to keep using it, or alternatively if we were to build custom software, what is difficult about upload that we would need to solve in a custom way to make easy?
-
-
Going back to your original post, anything that makes it easier is obviously better, but you do have to weigh that against the amount of work it takes to do it. I would rather that energy be spent on improving the AI for instance.
@LaFayette said in RFC - In-Game Map Uploads:
Question (1) - Do we want in-game map uploads?
What all would be involved in in-game uploads? Would that use the map maker? would it basically just be copying an already existing map, making your changes (like in edit mode) and then reupping it?
I could see that as possible being a relatively simple thing that just makes a new XML with different various things such as intial unit placements. Hopefully not too involved, but also not allowing for really impressive new maps like Frostion has been making.@LaFayette said in RFC - In-Game Map Uploads:
Question (3) - how should access control work?
You definitely need to have access for map admins for the official repository. There are too many maps that need periodic tweaks to leave it to people that might be gone a month after uploading their map.
@LaFayette said in RFC - In-Game Map Uploads:
Question (4) - should we keep github repositories and automate the map.yaml file?
What exactly does this file do in the engine? I presume it tells the local engine which maps are available and also which maps have new versions since you downloaded them originally.
If that is what it is, it seems to me (admittedly not a huge programmer) that it shouldn't be particularly difficult to do, but also doesn't seem like it does a huge amount. So lower effort, but lower reward.Personally, I think Github repos work pretty good. I can see ways that in game could be better, like for someone that just wants to change the XML (alliances, initial placements, initial territory owners, etc.), I think using edit mode to be able to make said XML could be very useful. If you want to go beyond that to use the engine as a GitHub front end, you are bloating the engine for functionality that can be found in other programs.
-
For the cost of 3 months to streamline uploading, I would like the time to be spent improving the AI, if it was I would;
-
Learn how to upload my maps.
-
Upload other peoples maps for a year and try and encourage others to up load their own using the current method.
LaFayette you listed the benefits below, this is what convinced me we should keep the map uploads in Github, basically as is.
all files available at a web link.
an automatic history of changes available at web links, this is critical for an editorial review of maps to verify what has changed
automatic packaging of files
capability to do bulk updates.
capability to do bulk searches. Having all maps checked out locally, it's some easy commands to search all XMLs in existence.
capability to do bulk file adds.
. -
-
@TheDog said in RFC - In-Game Map Uploads:
... * Upload other peoples maps for a year and try and encourage others to up load their own using the current method.
...I struggled hard with git. I still do, but I did figure out how to add a map with the help of others.
sourceforge you had to have someone upload your map. You have to have someone to do the yaml PR now as well, although my guess is that is less work than the sourceforge way, and it just notifies people that the map has been updated or added. Well, I don't think it notifies one when a new map is added but ...
At any rate, once your map has been uploaded, you can update it by yourself. It just won't tell users there is an update.
Have a thread for people who can't update/need help with their maps and those that know how, if they feel like it, can help teach how or just do it.
I thought triplea was trying to "decouple" a lot of maps, so until that gets sorted ... i guess someone still needs to decide what can and can't be.
-
@LaFayette said in RFC - In-Game Map Uploads:
What is difficult about repository upload exactly?
I can share my first experience I had with GitHub a couple of years ago. I tried desperately to find a GitHub turorial, googling "GitHub tutorial for dummies", but turns out I was more of a dummy than anyone dared guess. Some questions I couldn't find an answer to were:
- What is even a "repository"?
- What is this "commit" thing? Why can't I just change the files and call it a day?
- What is "pull" and "push"? What is the difference?
- If I have already committed changes, why is there still a need to push? Or pull?
- Etc.
See, the problem was that I had a mental model about repositories and stuff, but it was the wrong model. I tried to think of a repository as something like a folder. Multiple branches - multiple folders! Add a file to a folder - why isn't that file in the "same" folder online? If I merge a branch - how does it know which folder to keep and which to discard?
-
@LaFayette said in RFC - In-Game Map Uploads:
@RogerCooper A map validator is a highly requested feature. I've also been thinking it might make sense to automatically do hue shifting from existing images to fill in missing images rather than the hue shift be controlled by configuration (just make it automatic).
Some mods such as "The Grand War" use flags rather than hue shifts anyhow.
-
@ff03k64 thanks for the feedback, responding to the questions you asked:
What all would be involved in in-game uploads?
- The map uplaod process would be brougth in game, I think the map maker would gain a new button "upload maps"
- After clicking 'upload maps', you would be prompted to log in to the lobby (this then assigns your name so the backend knows who you are)
- You would select the map folder or zip to be uploaded
- Triplea would confirm the upload details and then you click okay and it would upload it.
- New map uploads would go into a moderation queue of some sort and have to be approved to be available for general download. This moderation would only be a content check and would not be checking that the map is playable or otherwise
- The map becomes available for download, probably from a default 'Experimental' category and we later deal with category promotion in some other way.
What exactly does this file [map.yaml] do in the engine? I presume it tells the local engine which maps are available and also which maps have new versions since you downloaded them originally.
Pretty much. When you click 'download maps', the game engine downloads this file and the data from this file is used to generate the map download selection. The engine uses the data from this file to know which version you are downloading and can later detect if there are new versions available or if you have the latest version or one that is missing altogether. The preview image and the map description you see in the download maps window comes from the 'map.yaml' file.
Hence, until the 'map.yaml' file is updated (actually called 'triplea_maps.yaml', but that is a detail), until that file is updated, game engines do not see the map as yet available for download.
I think using edit mode to be able to make said XML could be very useful.
The dream I think is for the front-end to be all encompassing for map making. I would like to even bring in the map making tools to be part of starting a game, so instead of an 'edit' mode there would be a 'map making' mode.
Though, using edit mode to change a map, be able to re-export that back to disk and then upload it is certainly the idea. The latter step, to upload, is arguably the most questionable because we don't necessarily have to build it (uploading is a generic problem, there are alternatives compared to us building a custom solution).
-
@TheDog Thank you for the response. I think I would agree pretty fully now that we would be unwise to drop Github repositories as our choice for map storage. It provides quite a lot, helpful in a number of ways beyond just storing the file data.
@beelee, turns out anyone can propose a change to the map yaml file:
- Head to the file stored in the engine repository, https://github.com/triplea-game/triplea/blob/master/triplea_maps.yaml
- Click the pencil icon
- The file will change from a read-only view to a text box that you can edit
- Make your edits, add a short description of what you changed & then click 'propose changes' (if you need to create a fork and do more github guides you through that and mostly does that work for you automatically).
This ties back into using github for map storage. The same pencil icon is available to edit game XML files as well, which means anyone can propose a tweak (or a correction) to any uploaded map file.
-
First a tangent. I responded to Alkexr's post above before i realized that they were talking about when they first experienced Github. It could still be useful to have a development glossary of some sort. Do we want that, or should I just delete what I typed? If we want it where should i put it?
So, back on topic. How much bloat do we want to put into the main TripleA download? Most people won't build maps. Would we possibly want a separate version that has any features like this that gets made? The canadian units already got removed because someone thought they were unnecessary and would help download speeds. At least that is what my memory tells me.
I think if we can get better documentation, that is probably more useful than all the work that would be involved with putting all this into the engine at this point.
-
I like the idea of having a thread to get users help. I wonder if that should be a full category even rather than a single thread? A pinned & locked post to that category could have a description of what to do, how to upload and have further links and details, all that would be needed to complete the upload process.
I tried to think of a repository as something like a folder. Multiple branches - multiple folders!
There are some source control systems out that work exactly like that : )
FWIW - A commit is essentially a file diff, a set of line changes and file additions and deletes. A branch is a list of commits. By "replaying" the set of commits git can reconstruct the state of all files to any point in the commit history. Also, this is why git never deletes as well because the contents must be available if you go back in history to before a deletion (so deletes in git are handled with tombstone markers that mark content as deleted, but it's not physically removed from disk).
The "main" branch by convention is called 'master'. To keep things simple we typically use just the one branch 'master' in map repositories. I think I'm seeing good usage for a really good explanation thread to
I think this goes back to having a help thread, explaining some of this, somewhere accessible, would probably make it quite a bit easier.
-
So, back on topic. How much bloat do we want to put into the main TripleA download?
It would all be source code changes. The cost would not be observed in download speed but instead for maintenance. If you build a house, you have to keep it up, make sure the roof gets replaced, make sure if you build a house next door that you don't disturb the foundations or cut the water main to the existing house. These are the costs we would see, just more code to get right and maintain (and future maintainers, whomever they would be, it's more stuff for them to read through and understand).
Most people won't build maps. Would we possibly want a separate version that has any features like this that gets made?
Agree, though making it as easy as possible I think is largely what TripleA is all about. I would like to try and keep the tools all-in-one so it's easier to find. Really, as mentioned above, I'd like to bring it all into the 'in-game' mode and have something like 'map making' mode that adds additional options to edit and save the map.
The canadian units already got removed because someone thought they were unnecessary and would help download speeds. At least that is what my memory tells me.
We (I) did an analysis of every unit image and which maps used them. Canadian units were used by something like 8 maps, and almost half of those maps had their own duplicative copy of the canadian unit images anyways. Out of around 130 maps, having 4 unique maps use images is very few. Images that are going to be in the common game-engine have a rule-of-thumb threshold that they ought to be used in perhaps at least 20% of maps (the WWII images are used in around 70%-80% of maps).
'
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login