RFC - In-Game Map Uploads
-
I've been working on a feature called 'map uploads' where instead of posting maps to github repositories, there would be an in-game UI where you could select a map folder or zip and then upload it. This is a really complex feature and I would like for us to really think it through and something that we would really like to have.
Historical Context
The First Map Repositories
The first map repositories were managed by admins & developers, usually one or two people. While this was simple for map makers, they only needed to email their map to someone, it had a number of problems. The lag time for when things were submitted and actually done was substantial, from weeks to months. It also creates pressure on that individual, they spend their time doing day-to-day operations (handling maps, uploading) instead of spending their time on other activities. We do not have very many admins or developers, we have a lot and very important development work to be done, thus spending time handling map uploads with a significant lag time is not a good situation.
Migration to Github Repositories
Github repos were introduced as a way to host maps and make the upload self service. Map makers, after a one-time initial setup could then upload and manage their maps from then on independently without an admin. This freed up the process bottleneck where an admin would have to be involved, saving labor on the side of the admin and eliminating the lag time of waiting for an admin to upload a map for you.
Problems with Github Repositories
Admittedly working in github is unpopular. IMHO, it is unfairly conflated with 'git', which is viewed as a toolset for developers, and that conflation makes it even more unpopular. Whether fair or not, there is non-zero interaction with git through github when dealing with map uploads, at the end of the day it is pretty simple, but when frustrations run high the minority have been vocal that github is not popular.
Concretely, the problem with github repositories is there still a small learning curve, buttons have to be clicked in a web form, an account has to be created, and there has to be the gumption to approach this. Even more concretely, the github web-ui does not do bulk file operations that well, if you want to upload more than 100 at a time you need to chunk it up, and if you want to delete more than several files it becomes tedious to do from the web-ui.
Map Uploads
When it comes to doing in-game map-uploads, we need to rebuild everything that github is providing us. When looking at the details, it's quite a lot! This list includes:
- file hosting
- file download
- change history
- ability to update the files
- access control, specific people and teams can be granted access to specific repositories
- map packaging into zips
- public visibility into all of the above
- quality control, all changes can be reviewed and vetted by admins to ensure that map policies are maintained
Map uploads would accept files from your file system and upload them to the TripleA servers and record their details in database. From then on it would be opaque, the only way to see the contents of a map would be to download it. There would be no way to know what changes have happened over time. Access control would have to be implemented from scratch.
Question (1) - Do we want in-game map uploads?
Is building an upload engine worth it? Git repositories do a lot for us. Are the problems really just getting over a short learning curve and bias against having to doing some additional work to get maps uploaded?
I've had some thoughts that in-game map uploads would be a game changer for people. With only a lobby account, you could upload maps and they'd be available for download.
Question (2) - Which of the features provided by github repositories do we want in map uploads?
Is it enough that you download a map, can then re-upload it and you overwrite the existing map? There would be in that scenario no way to know easily what changed, no way to browse map files through a web-UI (you'd have to download it, unzip it, and then view the files on your file system).
Question (3) - how should access control work?
Who should be allowed to edit which maps? The original uploader clearly should be able to modify maps, would they be the only ones who could modify those maps?
Question (4) - should we keep github repositories and automate the map.yaml file?
The triplea-maps.yaml file can be automated by a backend server. We can simplify the existing process so once a map is in github repositories, the server could detect a new version and automatically find the map download description, preview image, and then offer this as a download listing. At a minimum, we could eliminate and automate this file: https://github.com/triplea-game/triplea/blob/master/triplea_maps.yaml
Would simply and only doing this really be all that we would want?
Question (5) - do we want the upload to actually be a glorified front-end for github repositories?
We could structure the map uploader to essentially create a github repository for you and to commit the files into it. This saves us some work:
- no need to do file backups
- file contents are publicly visible
- change history is available through the git history viewable in github
We could decide to make this bi-directional so that if you are not the map author, you can do a map PR to the repository and the maps server would pick this up as an update. Otherwise any upload zips are then copied back into the repository.
The bi-directional integration I think is questionable:
- most map uploads have tended to be a one-shot, map makers upload and usually forget about it. They're not reviewing PRs into the repo's of the maps they have submitted, they're not even really looking at the repo anymore at all after the first upload
- map PRs are an existing thing and are falling through the cracks. We've a small list of map PRs that were never acted on and have just been hanging out.
So the question here, do we really need a front-end to view changes to maps, or is it just fine to only download the latest and just see where the map is and who cares how it changed most recently?
RFC - request for comments
This is a big feature - I really want this to do what we want:
- self-service map uploads
- something that we like having
- easy
- accessible to everyone
My hope this feature would drive for a LOT more maps to be uploaded. I really want us to think this through, if everyone could imagine how this would work, I'd like as much feedback as possible to help drive this to be the right feature (or perhaps not a feature at all if we think github repositories are actually just fine provided a person can grit down long enough to do the learning curve). I appreciate any comments/responses here, please consider this deeply as this is a major change to how maps are managed and will be the system we have for a long time to come.
-
i feel like anyone who can make a map should be able to deal with github that being said making things easier is almost always a win.
-
As a newbie to this, I don't think Git is particularly hard, though it could maybe use some better documentation. Not having to update the YAML would probably be an improvement though.
In my head, i feel like there should be an easy in between way to do it, but my head likes to think lots of things!!
-
Somebody needs to be in charge of the map repository and maintain some editorial control. Anything in the repository appears on the downloads screen for every user. The easier is to upload, the more problems there we be. Even now, we have problems with maps that don't run (usually because of missing images). We also need to be on the watch for duplicate names and TOS violations.
-
To one extent, part of the point is to encourage a lot of map uploads. As for editorial control, it's a problem in either process. I did not mention though the upload would need to go through a moderation queue so it can be verified to follow standards. Standards does not mean working, we should encourage sharing of maps no matter what state, but we have categories to represent the maps that are works in progress and those that are brand new vs the ones that are ready for main-stream play.
The editorial process has broken down a bit I suppose because we did not advertise the need for volunteers to be map-admins (it is a github group that can manage the maps). We also seem to have a lot of maps that have been shuffled around and have been in a broken state (and not in a proper "needs-work" category).
-
As a newbie to creating a map game there is a lot that one person needs to learn and do before a map game can be polished and uploaded to Github, in my case I dodged it and just hosted the file on my Google Drive.
Github game map hosting appears to be unnecessarily complex, I just want to upload so others can enjoy the map.
My question is, do you really need to have version control of all the individual files in a game map?
I hope the answer is no, because;
On Github could this work;
Uploading 3 files;
• Zip of the game map
• Png Screenshot of the map for triplea-game.org/maps-list/maps/
• Game Notes for triplea-game.org/maps-list/maps/The uploader would fill in a form depending on which option was selected, New, Minor Revision, Major Revision
• New - provide the 3 file above
• Minor revision – requires say 50+ characters to say what has changed and 3 files above
• Major revision – requires say 100+ characters to say what has changed and 3 files aboveIntended download access has two choices,
• TripleA Forum user ID required to download – this to encourage people to join the forum and to get feedback before being released to the public. This is a bit like me hosting on Google Drive as you have to read the post to get the link. To put a label on this its Experimental.
• Public access/releaseRequest to stop sharing a game map - only admins can do this.
Game map files are treated as a file set, so they are never deleted or added to or overwritten, except by admins.
There is a version control of sorts its based on the latest date of a file set and the zip file name.
Hopefully you get the idea.
. -
@TheDog Appreciate the feedback. TL;DR I wonder a bit if there is just a disconnect about how to work with github repositories. It turns out you don't have to do any zipping at all to either play a map that is in development or upload it. You can use flat-files in-the-game, and upload that exact same file structure. If you 'clone' the repository to your hard disk, it'll be 100% playable and you can modify the files directly and it's the right format for re-uploading.
=========== Longer response ===========
In some ways it should be easier to not zip your map. The game engine reads map files in an unzipped format, this matches exactly the format of a 'git clone'. You can clone the map repository, load the files downloaded into the game, play-test it, modify it, then re-upload those files to github without doing any zip work at all.
If we upload zips, then each change will be an entirely new zip file. We'll likely hit the 1GB repository limit quite quickly. Git never deletes anything. So if you ovewrite a 100MB zip 10 times, then you won't be upload anything further.
In part having the map be expanded is intentional so a person can browse the files in a map without having to download it. When you submit a PR to change a file, you see which file is being changed and it's not just an entirely new map zip (with maybe just one new file in it). If someone wanted to say change a single image, you can use the web-UI to 'add file' and submit the one image. If it were a zip, the person would have to download it, expand it, change the image, re-zip, then upload (and then it would not show what changed, just that the whole zip changed, and again would count towards the repo limit).
Png Screenshot of the map for triplea-game.org/maps-list/maps/
This screenshot is used in-game too when you download a map. Before you download a map it shows you a preview.
Game Notes for triplea-game.org/maps-list/maps/
Similar this is shared for in-game map download description. It has been typical to copy/paste game notes for this, but I would discourage this. Duplication is generally bad, it's also odd when there are multiple games each with their own notes, and finally when someone is reading about a map they want to know a blurb and not read the full game notes.
But I get the point that a map description, a preview, and the actual map are really the three things needed. Having at least the map description and preview be part of the map contents as a specifically named file could make things easier and would probably be necessary if we were to automate 'triplea-maps.yaml'.
• TripleA Forum user ID required to download – this to encourage people to join the forum and to get feedback before being released to the public. This is a bit like me hosting on Google Drive as you have to read the post to get the link. To put a label on this its Experimental.
• Public access/releaseThis level of control is non-trivial. I'm pro-advertising the forum in the right places, but I don't like having barriers put up for people.
Github game map hosting appears to be unnecessarily complex, I just want to upload so others can enjoy the map.
I'm not actually sure it's that complex, nor unnecessarily so. Having a working map is unnecessarily complex, the folder structure has to be just right. For github repository upload there is a web form and you drag and drop the files. I think the big disconnect is the realization you don't have to zip your files at all to play them in the game or to upload them.
As a newbie to creating a map game there is a lot that one person needs to learn and do before a map game can be polished and uploaded to Github
The intent is actually for work-in-progress maps to be uploaded very early, perhaps starting with just a single file. We have categories available for 'experimental' to indicate something is not working, and the map is not live until it is added to the index file 'triplea-maps.yaml'. Hence, you can work with unzipped files, upload them as you work on them, and the repository has a 'download' link that anyone can use to download a zip of your map. This automates a lot of what you would do with google drive.
Game map files are treated as a file set, so they are never deleted or added to or overwritten, except by admins
This is actually easier in github repositories as nothing is ever deleted in git. We at one point had speciifc versions of each map pinned in the downloads but it added some complexity and was removed to try and make it even easier on the map makers/uploaders. I'm starting to think there just needs to be more of a tutorial on how to build a map from one file, upload that, and then add more files to eventually have something that you can play in-the-game and have fully working. Though, the limiting factor is the game engine was not build to support multiple map versions, it assumes that any given game will have just one version installed. This is something that I think could be improved and each map version given a unique ID and then made permanently available. In this way the game engine could know which map it would need to download whenever confronted with a missing map or an older version.
-
I'd say the most positive aspect of having complicated map uploading process is preventing repository filling with even more unneeded maps and mods although I would still prefer having easier map uploading process with this conditions.
-
Giving priority to long lasting members at this area and considering their critics before uploading their projects.
-
Scoring new maps in this categories:
-
How distinct: the project compared to other similar era maps? I have to regretfully accept the fact that mods will almost never be played or be liked as much as their original counterparts. Most of their long lasting players already play the original ones because they already like it with that way. I don't even think mods can be more popular than the original ones even if they are absolutely better.
-
How Balanced: is hard to estimate. I'd say even having moderately balanced map is enough initially since it can always be rebalanced.
-
Replayability: The most important thing imho which can be achieved primarly increasing options and making all of them as good as with each other. For example I wouldn't want to see this kind of strategy guides in game notes: "X country should take A territory in r1, B territory in r2 and prepared to take C territory in r3. Y country is expected to hold/push D direction, Z country should avoid to doing E"
-
Graphics: Almost as important as the game itself. Like how good the relief tiles, unit arts, zoom rate, icons etc...
-
-
Round II
Here I am guessing how it could work, as I have never used Git for new maps;
I upload to Git my current game map Settlers: Age of Tribes it has 750 files, can I do this one upload?
I tell Git my map is based on a map called Age of Tribes, Git looks up the map and gets pointers to 216+216=432 files, the rest of the files are new, as most of the unit/flag png have been retouched by me although they look like alkexr pngs.
Scenario 1
Weeks later I have a change or too, but because Im human I don’t know what I have change, I upload all 750 files, I tell Git its based on Settlers: Age of Tribes, Git works out I have changed 3 files and stores them, the rest it already has.Scenario 2
I upload a new game map I am working on Settlers: Fallen Empire, it has 1200 files.
To aid Git, I tell Git it is based on;
Settlers: Age of Tribes (for the unit Icons and Flags) and
alkxer’s game map Fallen Empire (for the map tiles)
Git stores 5 files as they are unique to this game map.So could this work?
If so is it viable?
. -
@Schulz, how maps are uploaded and stored is independent of their ranking and categorization. Unless a map violates the map requirements, I don't think we should be telling anyone no.
I think your points speak to additional features to build on top that allow for both better manaul and automated ranking and categorization of maps.
It also appears that it's been unclear that we have been in need of more map administrators for a very long time and there is an open invitation to join. The map admins would help the upload process and fix & categorize the existing maps. The maps are like a garden, we've a lot of weeds growing.
-
If you use a GUI tool like "Git for Windows", or "SourceTree", you can upload all 750 in one go. If you are using the web UI that git provides, then you'd need to do 100 at a time. Considering it's a one time cost, I suspect that will take you about 10 minutes and then however long it takes to physically do the upload (so probably an hour total). For what is mostly a one-time cost, that does not seem unreasonable.
BTW - 750 files! That's a mega-map!
I don't understand how you mean by telling Git the map is based on Age of Tribes. If you are in a different repository, all the files are independent of any other map. It's the engine that has a concept of 'mods', but those need to live in the same map repository.
Re: Scenario 1
If you're using 'Git for Windows' or what not, it will tell you what you've saved on the file system. When editting XML, you can create checkpoints where you can go back and restore in case something gets messed up. This is actually a much better way to develop maps than saving a backup of XML every few changes (because you don't have to do so much manual saving, and git will tell you what has changed between checkpoints and allow you to restore to any checkpoint).
But fundamentally, if you re-upload everything, the files that are the same should just show no difference. I'll be a bit surprised if you don't know exactly what changed. You also should be able to upload any changes after you made them. If you're working on images for example, you can retouch a few, upload those, and then rinse-wash and repeat. Nothing says you have to do it in a big-bang.
Settlers: Fallen Empire, it has 1200 files.
Again, one time cost to upload it all into a new repository. Though, that is a lot of map files, where is that coming from? The engine does have limits, it is possible to that could have really poor performance or not be playable. Warcraft hero's taxes the engine and is barely playable. A map that is even larger might not have hope.
-
I think we need to refocus this conversation, but good input so far and please continue and consider this topic deeply. I do want to be sure we build the right thing. If we don't, then various problems we want solved will simply remain.
Main Questions so Far
-
Is Github repository upload actually that complicated, or is it that there is a small learning curve that is hard to approach?
-
Github repository allows anyone to change a single file or many - is this something we value? In other words, is a targeted single file upload valuable, or are we okay with any change requiring that the whole map be downloaded, extracted, changed, re-zipped, and then re-uploaded.
-
How valuable is multiple person collaboration? If everything is done by one person who is the original map author, and we are okay passing zip files around via google drive, it's okay. Likely permissions to update a map will be restricted to map-admins and the original uploader. It seems like not being able to 'pass-a-map' forward would be a problem and adds a lot of scope to what would have to be built for map uploads. At some point we are building something so similar to a github repository that one wonders if we would be better to just focus on making the upload process easier to follow.
-
-
Github repository is probably not the tool for the job, it might have a small learning curve, but I am still not persuaded to use it, it requires another tool to upload a game, a quick search of the games I have and the smallest is 450 files.
Github is just an obstacle that I dont want to over come as it offers nothing that I want to use, however the restore points for an xml might of tempted me, but no, I will stick with my many backups and comments in my xml files.
.As mentioned I would like to just upload 3 files, as 400+ files takes too long.
But the process is
zip, upload
change, zip, upload
But, there is is no download, extract as only one person will be doing the changes.
.As for collaboration I would ask outside of Github for help, email back and forth, share on Google Drive , so no need for Github. Game maps tend to be by one person, that's another reason why Github does not suit one person as its really designed for collaboration.
.
.
ps. the Fallen Empire has 416 base and 416 relief tiles, yes its a big map, it was design by alkexr, but never finished, the rest of the files I need to tidy up, but it runs well as a single player. -
Github repository is probably not the tool for the job, it might have a small learning curve, but I am still not persuaded to use it
I'm not trying to pursuade you to use it. If you want your maps uploaded, it's what you have had to do. Maps have always been stored in source control (previously sourceforge). The biggest difference is the work is now self-service (relatively easy) and development does not manage it so closely. This frees everyone up.
-
If the 100 file limit is a bottleneck then you can use a desktop GUI that makes it easy to do the needed git commands locally and then push all files at once.
-
Github enables collaboration, but I use it personally for solo projects as well and am aware of many other examples. The email'ing, sharing via google drive, it all could be removed by commiting to the repository instead.
-
I think the solo led collaboration is common but it's not always the case. The really popular maps have many people working on them, particularly over the years (think long term). For example we've had maps where the unit images were upgraded.
-
There are more benefits to having github repositories, for example all XML is available at a web link, makes it really easy to see how other maps were built and examine them without downloading and exracting. It also makes it easier for admins to go in and correct issues, and if used would allow map makers to link to their XML and have others look at it easily.
-
-
I'm starting to lean towards the thought that storing map zips in database only would be a mistake. The repositories provide a lot of value from different perspectives:
- all files available at a web link.
- an automatic history of changes available at web links, this is critical for an editorial review of maps to verify what has changed
- automatic packaging of files
- capability to do bulk updates.
- capability to do bulk searches. Having all maps checked out locally, it's some easy commands to search all XMLs in existence.
- capability to do bulk file adds.
Diving into some of those details, we have built an in-code test that downloads nearly every XML that exists and verifies that the map parser is able to read them without any errors. If we mess up the map parser somehow in an obscure way we've a good chance of finding that. Every time we change code, we run this test automatically as a requirement for that change to be accepted.
The file history is automatic and really critical for editorial review. If we were to have zip files only, then the already nearly non-existent map-admin team would then be slammed with having to open every new zip file uploaded and fully verify the entire map rather than having a web link tell them exactly what changed.
With all maps checked out, it's possible and has been done to bulk change every XML that we have to apply corrections. Similarly being able to search all files has let us find features which are not used, or find features that are heavily used or lightly used, get statistics on which images are duplicated the most and least, and generally it's provided useful insights that would not be possible if we had to download from scatch every zip and extract.
File adds is where repository does shine. Consider adding a standard config file to every map. That is a very likely scenario coming up as we break up and distribute the index 'triplea_maps.yaml' file to each map (which then avoids needing to have the single centralized index file). So for example, we add this config file to every map: If every map is in database and uploaded as a zip, we need to coordinate with every single map maker such that they add the file or download the latest zip and merge. If we fail and a map maker uploads a zip without the config file then they'll inadvertently remove it. Checking every map consistently keeps the config file will become non-trivial and painful and a source for troubleshooting and error. Overall, a messy scenario that incurs more work than we are doing today.
Getting to the point, github repositories provides: file storage, version tracking, access control, web hosting & automatic packaging. That's quite a lot to be getting for free and would be an enormous amount to build ourselves from scratch. Perhaps making bulk uploads easier with good step-by-step instructions is a lot of what we are missing.
-
@LaFayette You need to verify for missing images. My experience is that is the number one problem with maps, including 2 in the current repository.
-
I had easier time learning to create a map from scratch than understanding Github sufficienly in spite of Frostion's guidance. After that it is easy to manage and of course very valuable if map user can achieve to do it.
Would Bitbucket be better alternative?
I think nobody should be able to in charge of any project other than the original creator besides fixing bugs.
-
@RogerCooper I found another non-working map in repository, "Caravan Plain".
-
@RogerCooper A map validator is a highly requested feature. I've also been thinking it might make sense to automatically do hue shifting from existing images to fill in missing images rather than the hue shift be controlled by configuration (just make it automatic).
-
@Schulz What exactly did you find difficult in Github? To me the web UI is as easy as it could almost be. It is almost literally click a button, drag and drop files, then click another button to submit the changes. Hence, I'm very interested to know where the complexities are.
No, BitBucket would not be a better alternative and it has none of the web UI features, you would have to 100% use a Git client that you install on your computer and then BB has other issues. We are not migrating hosting system.
