Pages: 1
  Print  
Author Topic: Migrate Releases  (Read 3000 times)
Offline (Male) Goombert
Posted on: September 02, 2015, 01:49:20 am

Developer
Location: Cappuccino, CA
Joined: Jan 2013
Posts: 2993

View Profile
I want to propose this since I've been making more use of it in recent projects. When we archived the old LGM versions we stuck them in our dropboxes which is, despite me having created a shared ENIGMA dropbox, still subject to link rot. I never did attempt to migrate our old and new releases to GitHub because I never understood the feature until now.

Basically if we move all of the ENIGMA ZIP's, old LGM jars, and all those old revisions and packages over to the GitHub release pages we get static links.
For example:
https://github.com/IsmAvatar/LateralGM/releases
https://github.com/IsmAvatar/LateralGM/releases/download/1.5.7.1/lateralgm1571.jar

Instead of what we have now on the LateralGM: Revisions Wiki page:
https://www.dropbox.com/s/jrigtjk0yld8zsy/lateralgm1571.jar

Notice how the GitHub version is not subject to link rot? This is by GitHub's own design for the very reason we are intending to use it. Using the GitHub release pages also allows me to archive the change logs in the description. This is important because our forum is also subject to link rot as topics are moved and the forum board evolves, people will still be able to find the change log well into the future (considering GitHub is probably more likely to be around in 10 to 20 years than we are). This also means we don't have to make so much use of third parties like dropbox since setting up proper file transfer to ENIGMA's server would be a lot of work and cumbersome to say the least (which is why we are still struggling to finish the EDC). This also means that every contributor can establish a release if they have appropriate access to the repository.

There are some things that need addressed. Primarily the GitHub release mechanism automatically versions the repository and detects change logs. So if you don't enter a description it automatically detects the changes made to the repository since the last release was made, which we can make more effective use of in the future. This will make our initial migration of the old releases cumbersome though as I'll have to paste all of the change logs there and you'll just have to ignore the one GitHub shows in cases where a change log can't be found (I have them for all of my releases just not LGM 1571), which is not that big of a deal. This also means however that ENIGMA needs to establish version numbering and so do all of the other projects including JEIE, etc etc.

Once this is done however I would update all the appropriate documentation on the Wiki and related links. I would not delete the old revisions archived on our shared dropbox but the Wiki and everything would be updated to reflect the new release process. I also would not delete the old revisions page right away until some point in the future when the links become broken or our shared dropbox has been fragmented.
http://enigma-dev.org/docs/Wiki/LateralGM:_Revisions

I would continue to maintain the Extra Packages page for quick links to all of the latest releases. The page would just be updated with the rest of the Wiki.
http://enigma-dev.org/docs/Wiki/Install:Extra_Packages

I've experimented with archiving two old releases of LateralGM on its repository.
https://github.com/IsmAvatar/LateralGM/releases

I also believe this would reduce the confusion of random people finding LateralGM through Google and not knowing where to download it. The GitHub release mechanism is pretty common and you know people have come to get used to it, myself included. It also makes formatting the change logs nicer by allowing us to use markdown and link to specific tickets instead of by URL like I do now.

I'd like to know what you all think about this, I think this is the way to go and I am looking to do all the footwork to make the change. I'd just like to take some feedback and suggestions on it first.

Also I will need additional privileges to a few repositories such as JEIE to do the releases which I asked IsmAvatar about before but never got a response, hopefully I can get a hold of her.

Edit: Actually it seems like it is possible when drafting the releases to base them on a certain commit, but I don't know how far back we'd have to dig. I could probably dig for all of the releases I made but maybe not the ones for 1.6 and 1.5 as they may not have been migrated from Sourceforge.
« Last Edit: September 02, 2015, 02:17:35 am by Robert B Colton » Logged
I think it was Leonardo da Vinci who once said something along the lines of "If you build the robots, they will make games." or something to that effect.

Offline (Unknown gender) TheExDeus
Reply #1 Posted on: September 02, 2015, 01:56:21 pm

Developer
Joined: Apr 2008
Posts: 1860

View Profile
I totally agree with this. I haven't made GitHub release myself, but I have used them and I like it. I think it is super cool if it allows automatic changelog creation, as currently I do it manually by looking at commits.
Logged
Offline (Male) Goombert
Reply #2 Posted on: September 04, 2015, 03:13:27 pm

Developer
Location: Cappuccino, CA
Joined: Jan 2013
Posts: 2993

View Profile
Alright Josh is good with it. As a first experiment I've moved all of the LGM releases over there, it was a total pain digging up the old commit logs but I did the best I could. Best thing to do is just to make sure we do it every future release.

https://github.com/IsmAvatar/LateralGM/releases

I linked the forum posts with the change logs instead of putting them into the release description because most of them are super long which makes browsing that releases page a bore. I will continue to do this for future releases but may add some small additional details. From now on each new release of LateralGM and major build will be a new release on GitHub and a new forum post, so it won't be one long forum post as was done in the past.

I'll do the plugin jars next but I don't think I'll dig up the specific commits for them, I'll probably just base them all on the most recent commit then do it correctly for future releases. I believe we should continue to version the plugin against the LateralGM version but I'm not sure yet.

If we are going to do this for the ENIGMA portable zip then the ENIGMA tracker also needs to work out a proper version number and we need to start incrementing it to mark the releases.

So basically we can use the tags for marking major milestones or w/e and the releases can be generated for each major milestone/revision. Does everyone like this?
« Last Edit: September 04, 2015, 03:22:11 pm by Robert B Colton » Logged
I think it was Leonardo da Vinci who once said something along the lines of "If you build the robots, they will make games." or something to that effect.

Offline (Unknown gender) TheExDeus
Reply #3 Posted on: September 04, 2015, 03:40:08 pm

Developer
Joined: Apr 2008
Posts: 1860

View Profile
Maybe we need to add a changelog in the repo as well, so that when a person downloads the release from github he can still see what has changed (even if the forum dies for example).
And how do we start versioning ENIGMA? Just start at 1 or something? I personally think ENIGMA is working enough to actually be considered version 1 right now (except the LGM crashes). And then just go from there. I kind of liked smaller versions previously (like 1.21, 1.22, 1.3 etc.) but it seems the current trend for a few years now is just increment the version by 1. So we get firefox 40.0.3 now and so on. So the after dot revisions are bugfixes and almost every feature is a major revision.
We should also create milestones, so we know when to release instead of on every commit.

Also, can github create a release for the current master automatically? Like instead of a specific version release the top one is always the current up-to-date one (like "beta" releases). I guess all of that must be done manually?
Logged
Offline (Male) Goombert
Reply #4 Posted on: September 04, 2015, 10:26:34 pm

Developer
Location: Cappuccino, CA
Joined: Jan 2013
Posts: 2993

View Profile
Quote from: TheExDeus
Maybe we need to add a changelog in the repo as well, so that when a person downloads the release from github he can still see what has changed (even if the forum dies for example).
The problem with this is that if the release log is verbose/long then it is really hard to browse for other releases. The simple thing is to say to just make the release log shorter but that's not always feasible. The other solution is to make more frequent releases so the logs stay smaller. Personally I think GitHub should improve the releases feature by making the browse part short descriptions then clicking on the release gives a change log and a longer description, maybe I'll file an enhancement request/suggestion. Though I doubt they'll care what I think since they don't even listen to Linux Torvalds who created git.

Quote from: TheExDeus
And how do we start versioning ENIGMA? Just start at 1 or something? I personally think ENIGMA is working enough to actually be considered version 1 right now (except the LGM crashes).
I believe ENIGMA already had a version process and we're technically still working on r4... This would be a question for Josh. Josh has stated in IRC that he has old copies of r3 that should be able to run which I'd like to see archived on GitHub releases page with the newer portables just for historical purposes.

Quote from: TheExDeus
I kind of liked smaller versions previously (like 1.21, 1.22, 1.3 etc.) but it seems the current trend for a few years now is just increment the version by 1. So we get firefox 40.0.3 now and so on. So the after dot revisions are bugfixes and almost every feature is a major revision.
I too like smaller revision numbers in case you didn't notice we never got to LGM 2.0 yet. I think Josh also tried to persuade me into smaller version numbers before.

Quote from: TheExDeus
We should also create milestones, so we know when to release instead of on every commit.
I forgot about the milestones API.
https://guides.github.com/features/issues/
But there's also the difference in annotated vs. lightweight git tags. Tags are built into git, not a GitHub feature, releases is a GitHub feature. Releases have a 1 to 1 correspondence with tags and tags have a many to 1 relationship with commits. A lightweight tag just references a commit "git tag -l v1.2 a1a1a1a" where a1a1a1a is the first 7 alphanumeric characters of the commit hash/number. Annotated tags can reference other objects or you can create them when committing using -m to enter a message or w/e and say something like "This commit marks the completion of v1.2"

The fact that tags and releases are 1 to 1 really screwed with my head at first, this means you have to make a new tag for every new release, you can't reuse tags. I really found this to be quite odd, so the direct links will always be a little redundant.
https://github.com/IsmAvatar/LateralGM/releases/download/v1.8.6.844/lateralgm186844.jar
See what I mean? We could just rename the jars to just plain lateralgm.jar but I don't like that because then you can't download them all at once easily, you have to download then rename download then rename etc. Open to changing this though because it would make it easier to include in the packages file in the one ENIGMA repo that it uses to update on Linux using that script, which has to be changed to include the updated hash of the jar anyway to see if it's newer. On the other hand I should mention that nobody else on GitHub uses the same file name for every release like I just proposed.

Quote from: TheExDeus
Also, can github create a release for the current master automatically? Like instead of a specific version release the top one is always the current up-to-date one (like "beta" releases). I guess all of that must be done manually?
Yes, that's how I did the last 1.8.6.844 release you just create the tag on GitHub. I used the command line to create the tags for the really old commits because GitHub only shows a list of recent commits to create a tag. Just draft a new release, open the "Recent Commits" under @target and select the top one.

Now the other thing to mention is that I lied about GitHub creating change logs. This is really stupid because given both commit hashes/numbers you can do a diff from the command line with tags, GitHub really sucks at these things (why Torvalds does not merge pull requests from GitHub). However other people have invented some automation tools that generates change logs which could be integrated. But also as I said above I don't like putting the long change logs on the releases page because of how terrible it makes it to browse for an old release, this is why I plan to stick to small change logs for each release with a brief description and the forum post containing the longer change log for the foreseeable future, unless you can think of some better way to do it. I do agree with you that I'd rather have it on GitHub though where it's more permanent.
https://github.com/skywinder/github-changelog-generator
Logged
I think it was Leonardo da Vinci who once said something along the lines of "If you build the robots, they will make games." or something to that effect.

Offline (Male) Josh @ Dreamland
Reply #5 Posted on: September 04, 2015, 10:38:09 pm

Prince of all Goldfish
Developer
Location: Pittsburgh, PA, USA
Joined: Feb 2008
Posts: 2949

View Profile Email
It's true that I never tagged a release as r4. Things kept changing—more than anything, my expectations. You could number from there, or you could just call it V1. Or V1 RC1 and have fifty of those before you just call it V1. It's up to you. I prefer rolling releases. The way version control is supposed to work is that master is supposed to be stable at all times; ready for a nightly build whenever. You then have multiple work branches that might not build, or might not work. This never really came to be in ENIGMA... my attempts to introduce it ended badly.

GitHub can tag releases for you, yes.
Logged
"That is the single most cryptic piece of code I have ever seen." -Master PobbleWobble
"I disapprove of what you say, but I will defend to the death your right to say it." -Evelyn Beatrice Hall, Friends of Voltaire
Offline (Male) Goombert
Reply #6 Posted on: September 04, 2015, 10:44:42 pm

Developer
Location: Cappuccino, CA
Joined: Jan 2013
Posts: 2993

View Profile
Josh can you share the old ENIGMA copies with us if you have them on hand for me to archive on GitHub? I would also like to see the last ENIGMA zip I had on the shared dropbox up there too, so maybe we should call that v1 since it serves as an intermediate milestone to the latest version from r3. I like to keep archives of old versions because it helps with testing.

Also the new tags on LateralGM make it really easy to see what the master looked like for a particular release, which is one really good thing about this. I do not think I care much for milestones but still think we should use them, they are just watered down labels but they still help abstract the project progress.

Second, I too favor rolling releases because if we did automate the change log generation we could just make it against the most recent commit/PR in master. I don't like all of these granular changes and I've been working on making more inclusive PR's/commits. The real hard problem with this is that I try to fix one issue at a time and push it to GitHub to see if it fixes a problem. I don't like sending a large amount of commits with a fix for something only to later find out it didn't actually fix it. This is difficult because people do not like to test or help test, I am not referring to us, I am referring to users that report issues.

FYI: Maybe we don't even need change logs Harri with the GitHub compare feature.
https://github.com/blog/612-introducing-github-compare-view
https://help.github.com/articles/comparing-commits-across-time/
https://help.github.com/articles/comparing-commits-across-time/#comparing-tags
« Last Edit: September 04, 2015, 11:38:02 pm by Robert B Colton » Logged
I think it was Leonardo da Vinci who once said something along the lines of "If you build the robots, they will make games." or something to that effect.

Offline (Unknown gender) TheExDeus
Reply #7 Posted on: September 05, 2015, 08:20:05 am

Developer
Joined: Apr 2008
Posts: 1860

View Profile
Quote
The problem with this is that if the release log is verbose/long then it is really hard to browse for other releases. The simple thing is to say to just make the release log shorter but that's not always feasible. The other solution is to make more frequent releases so the logs stay smaller. Personally I think GitHub should improve the releases feature by making the browse part short descriptions then clicking on the release gives a change log and a longer description, maybe I'll file an enhancement request/suggestion. Though I doubt they'll care what I think since they don't even listen to Linux Torvalds who created git.
That is not what I meant. I meant adding a changelog.txt in the repo so that when the person downloads ENIGMA.zip he has that inside the ENIGMA folder. This way he can always see what has changed (and there could be a million changes all we care) and yet the GIT releases are a lot more concise and don't show all the changes like you want.
Logged
Offline (Male) Goombert
Reply #8 Posted on: September 05, 2015, 08:54:49 am

Developer
Location: Cappuccino, CA
Joined: Jan 2013
Posts: 2993

View Profile
I'm not so sure about that then Harri, I don't really think I like that idea either. I support the idea of distributing the change log with the release inside the ZIP, but I see no reason to add it to revision control, we would literally be revising our own revision logs (is this not what git should be doing for us?). I fully support people's ability to use ENIGMA or any software offline, even if you have internet there's many parts of the world that don't including a lot of public parks and such where it's nice to stick software on a thumb drive and away you go. I don't even know if you can get internet/wifi on a plane or not yet cuz 9/11. Anyway I did think that maybe we could set up that git change log generator project and make it auto commit with one of the many bots we have to a repository that contains change logs for all ENIGMA projects. But I didn't like that idea because of the same reason I don't like you suggesting it.

I always did the change log as a more human readable form of the git revision log for those who may not know how to use git. Because of my historic inability to make nice commits I've also used both github's revision history and the change logs I've made to find stuff.
Logged
I think it was Leonardo da Vinci who once said something along the lines of "If you build the robots, they will make games." or something to that effect.

Offline (Unknown gender) TheExDeus
Reply #9 Posted on: September 05, 2015, 07:14:50 pm

Developer
Joined: Apr 2008
Posts: 1860

View Profile
We often don't have nice list of changes in the commit logs either. It is usually just named "fixes" or something with only the most grand change added in the title. I have been trying to make the changes a lot prettier though with listing. Like all the commits in the past few years from me have been in a form of:
-New feature
-Change
-Bug fix
And so on. Sometimes I include additional information so some of the bullet-points can get larger. In a log that would not be needed.

Quote
but I see no reason to add it to revision control, we would literally be revising our own revision logs (is this not what git should be doing for us?)
As far as I know git can only generate a log with "git log --oneline --decorate --color". The log shows only the commits and their titles though. Not the full message. So this would only work if we do every single small change in a separate commit. That is something we usually don't do and I'm also sometimes doing so many things in parallel a single commit can have 40 changes.
And I don't agree with the "we would literally be revising our own revision logs" as we are still doing it manually anyway. If you want to include a changelog in a release, then you will have to manually assemble it. And for each release add the new changes. If you are going to do that process anyway then why not include the .txt in the git repo? Then we could at least automate building of installer and you wouldn't loose it.

So the automagic generation of logs is only possible if create one commit per change. And git committing is actually something we haven't discussed anyway. We have some kind of style guide for source, but how about style guide for git? Like:
1) Every change in a separate commit.
2) Change must be described in title of commit message.
3) More details or notes can be done in description of commit message.
4) Don't bundle several changes together in one commit.
5) Master should work (compile and run) after each commit.

I would personally not like these points, as I cannot guarantee (5) while working in a branch and I usually can't guarantee (4) and (1) because sometimes I can't separate changes. Like I add 5 new functions in GSstdraw.cpp. I can't make 5 commits for that file unless I commit after each function add. That slows down development.
Maybe Josh has some interesting ideas here? Like how is he used to work in G or something? I think they use Git too.
« Last Edit: September 05, 2015, 07:17:02 pm by TheExDeus » Logged
Offline (Male) Josh @ Dreamland
Reply #10 Posted on: September 05, 2015, 07:44:36 pm

Prince of all Goldfish
Developer
Location: Pittsburgh, PA, USA
Joined: Feb 2008
Posts: 2949

View Profile Email
There's hardly a more descriptive change title than "Hacks in a fix for multidimensional array subscripts. Fixes #904." The Git spirit is small commits with simple change descriptions. Coming from Subversion initially, I prefer larger commits with more detailed messages, to which git is also conducive. The list you gave essentially embodies the spirit of source control.

Google does not use git itself. Google3 is a completely linear, monolitchic codebase. It is not possible to commit code to Google3 that does not build, unless you issue commands to bypass the mechanisms in place. If you do that, you had better know what you are doing.

Anyway, it's clear to me that you know what these commit messages should look like. You may find tags or merge commits helpful. That is, do your work in your own branches with lax commit messages, and then either flatten them for commit to enigma-dev/master, or merge them and specify a useful commit message. You might consider annotating these with some kind of searchable token so that you can create changelogs between versions automatically. Following this, even if you were not careful to annotate all merges to master, you could still crawl all commits for useful changelog points. This could be as simple as putting Changelog point: before each line in a commit description that actually contains noteworthy text.

Just some ideas.
Logged
"That is the single most cryptic piece of code I have ever seen." -Master PobbleWobble
"I disapprove of what you say, but I will defend to the death your right to say it." -Evelyn Beatrice Hall, Friends of Voltaire
Pages: 1
  Print