Pages: 1
  Print  
Author Topic: Website and IRC Problems Version Control Could Fix  (Read 3615 times)
Offline (Male) Josh @ Dreamland
Posted on: May 26, 2013, 10:48:36 am

Prince of all Goldfish
Developer
Location: Pittsburgh, PA, USA
Joined: Feb 2008
Posts: 2949

View Profile Email
Sorry to add yet another topic to this already overpopulated board. Maybe one day the whole forum will be this active, and I won't have to feel bad. I figure this is a good board to pollute, since people are already paying extra attention to it.

To cut to the chase, we need to start using version control EnigmaBot and other features of the website.
I have organized my points into •B•U•L•L•E•T•E•D• •L•I•S•T•S• to save everyone some brainpower and reading.

Rationale:
  • Security: We don't have anything to hide except password hashes and private messages, which are part of the SQL database and would not be versioned.
    • Moreover, we have tons of living proof that being open-source promotes contribution of security patches over exploitation of security holes (consider Linux).
  • Feedback: This lets people drop pull requests and submit bug reports for features of the website that irk them.
  • Disaster Recovery: This makes site-wide code backup obsolete.
  • Ease of development: This makes updating pieces of the website a no-brainer, lets users edit code in their own environments.
  • Peace of mind: This removes fear of fucking something up on the server and having to dig for the backup manually (yay, git checkout).
  • Robert: I'll never have to put up with another episode of "zomg this needs blue and yellow and a grid background, so shell into the server and install my CSS patch or I'm leaving"
  • Progress: Ism and I don't have to be the only people scratching our heads at why the fuck EnigmaBot has all these fucking issues
    • We can take care of separating sensitive information to produce redistributable code up front, once and for all.
    • Someone else can deal with the fact that our current SMF theme is based on ten-year-old code.
  • Appropriateness: We version a lot of dumber projects, Calico being the dumbest to-date.
    • Other people and communities could benefit from the shit I had to go through to hook the EDC up to SMF's database

The idea: EnigmaBot and the SMF/Wiki extensions we have written need placed under version control, possibly in the same repository, at some point down the road. 

Problems we need to address with which this change may or may not help:
  • EnigmaBot:
    • When operators join, they are not notified if they have messages.
    • Some of the repository search features are screwy
      • Something is seriously wrong with !egrep / !lgrep / !etcgrep. It often completely misses search results.
      • !grepnext cannot skip to the next file, in cases where there are nine hundred results in three files. The user is told only that there are 900 results.
      • !eurl is a bit inconvenient; it does not attempt to find a file of near match (eg, by adding .h or .cpp)
      • !eurl can not fetch the next file with a matching name, or a file with a matching name in a matching folder. Try !eurl'ing for SHELL/Makefile.
    • This hasn't been a problem lately, but if an SQL timeout occurs, the bot's next query will fail relatively unceremoniously, but then the connection will be re-established and subsequent commands will succeed. Why can't the current command succeed, too?
  • Forums:
    • The forums are using an SMF 1.x theme, despite being on a (now outdated) version of SMF 2.
    • One (very bitter, but still possibly correct) ex-user reported that this is a violation of the license used by SMF 1. I'll need to do more research on this.
      • I believe that the license only concerns code redistribution (which is a problem if we want to version this forum software!), but,
      • we may well need to update our footer code to include an SMF notice (which I don't have a problem with).
      • My license preference is GPL, so if possible, I would like to have our extensions licensed under it
    • A lot of good features are missing from our theme, due to its age and the consistency of our code at the time (Gary was 17, I 16, a2h 15).
    • The icons are a bit inconsistent, and aren't vectored (I also made them when I was 16).
    • SMF code and the extensions folder will not be versioned. If you want an extension added to SMF, file a tracker artifact.
      • The actual BBCode extension, multiple-verification-answers extension, and any other extensions unique to or maintained by the ENIGMA team will be versioned.
  • Site:
    • The header and footer are dated (as in old), outdated (as in incorrect), and ugly (as in ugly).
    • The transition to a2h's "v5" design will go much smoother in the presence of version control
    • We could really use site scripts to handle release control and package builds
      • The server could package the release zips currently maintained by Robert and Polydip
      • Users could vote (yes/no) whether a release package/zip works for them, and the zips/packages with the highest rating could be listed first.
  • EDC:
    • Written when I was 18 (what does this tell you?)
    • The badge system is not implemented, because I didn't understand how M:N relationships worked in SQL (comments seemed wasteful to me, and badges? Forget about it).
    • The download system is also not modular (you can only give one download link!)
    • Screenshots are unimplemented.
    • There's no way to view games and blogs by most recent activity (one of the things the forums would offer over the EDC for posting games)
    • You have to use external hosting for your games (meaning broken links will abound; cloud hosting is becoming extremely cheap compared to when I was younger
    • Virus flagging and game rating are not implemented. These are absolutely necessary before we start our own hosting and accumulate a large userbase.
    • If we do our own hosting, a file manager will be necessary.
  • Wiki:
    • Either the extensions folder will not be versioned, or pull requests to it will not be accepted. If you want a Wiki extension added or updated, file a tracker artifact.
    • There may or may not be problems with the EDL and YAML GeSHi plugins, both of which were written by me (I know there are problems with the YAML one; I'll fix those at some point).

Writing this has given me carpal tunnel.

Ism and I will get everything moved into a git repository and checked for sensitive information/passwords/etc at some point in the discrete future following discussion here. Any additional problem you have with the above, list them, as they are likely to be dealt with long before the site is on GitHub.
Logged
"That is the single most cryptic piece of code I have ever seen." -Master PobbleWobble
"I disapprove of what you say, but I will defend to the death your right to say it." -Evelyn Beatrice Hall, Friends of Voltaire
Offline (Male) polygone
Reply #1 Posted on: May 26, 2013, 11:48:54 am

Contributor
Location: England
Joined: Mar 2009
Posts: 794

View Profile
I want to be able to link to 64digit games on the EDC.
« Last Edit: May 26, 2013, 01:12:31 pm by polygone » Logged
I honestly don't know wtf I'm talking about but hopefully I can muddle my way through.
Offline (Male) Josh @ Dreamland
Reply #2 Posted on: May 26, 2013, 01:11:57 pm

Prince of all Goldfish
Developer
Location: Pittsburgh, PA, USA
Joined: Feb 2008
Posts: 2949

View Profile Email
W...what?
Logged
"That is the single most cryptic piece of code I have ever seen." -Master PobbleWobble
"I disapprove of what you say, but I will defend to the death your right to say it." -Evelyn Beatrice Hall, Friends of Voltaire
Offline (Unknown gender) forthevin
Reply #3 Posted on: May 26, 2013, 03:27:34 pm

Contributor
Joined: Jun 2012
Posts: 167

View Profile
I think this sounds like an excellent idea, and the plans seem sound overall.

Forums:

In regards to the license, version 2.0+ of SMF uses a 3-clause BSD license, which is compatible with GPL3. As far as I can tell from the description on Wikipedia as well as the license description (http://www.simplemachines.org/about/smf/license.php), redistribution of source code or software license with the Simple Machines License, as well as any modifications, requires written permission from SMF. The themes from SMF 1.x are licensed under the Simple Machines License, but I don't think it would make any sense if the webpages generated using a theme are also licensed under the Simple Machines License, since then even basic usage of SMF would require written permission. Therefore, given that we are not currently, and do not plan to, redistribute either SMF or any themes for any version, or any modifications of those, I am convinced that there are no issues.
That said, in the long-term it would be nice if we could upgrade the theme to 2.0+, but I am convinced that there are no problems with the current plans.

Site:

Excellent plans. I have personally thought about adding automatically run tests to ENIGMA, primarily as an aid to development. I also think some of those tests could be run for releases, to see statistics of the tests (how many passed, how many failed, etc.), and possibly reject releases that do not work at all. A tests page could also be added to the site, and tests run daily/weekly or something like it, and statistics shown for each run, such that developers could more easily track regressions in bugs and see which bugs have been solved.

EDC:

Good points in general.

In regards to the download links, it would be useful to have platforms for each download link. I also think it would be useful to have some sort of option for playing the game directly on the page or on another page instead of downloading, when we get HTML5 support. That could be very nice. An option for downloading the source would also be very nice.

I think it will definitely be necessary to be able to search for games. I also think adding tags to a game, such that users can search for names and/or certain tags (such as genres), is a good idea.

An updating/reupload mechanism for games may also be a good idea, in case the user adds more content or fixes bugs.

In regards to viruses, there needs to be a mechanism at some point for "moderators" to remove games that contain/are viruses. I don't think it is needed in the start, but as the EDC grows, I think it will be necessary in order to scale.

And polygone is being silly, Josh is confused, and Robert says he is happy today.
Logged
Offline (Female) IsmAvatar
Reply #4 Posted on: May 26, 2013, 03:35:11 pm

LateralGM Developer
LGM Developer
Location: Pennsylvania/USA
Joined: Apr 2008
Posts: 877

View Profile Email
While I certainly don't oppose the idea, I do, however, feel like this is just another case of "stick it on github and the problem is magically solved". The problem with this ideology is twofold: 1) Interested manpower. Most of our interested manpower is already working on the wiki and enigma, which have higher priority than the bot and the website (as they should). As for generating interest, that leads to point 2) Generalizability. The bot and the website are pretty much singletons. They don't generalize to anything else, so there's not really a need for a second one to be launched anywhere else on the entire internet. Generally a project is best suited for source distribution and version control when it's expected that there will be multiple instances of it.

This leads to a more serious issue: more leechers than seeders. With so few people actually contributing to these projects, the only thing opening the source code effectively does is allow people to read it, learn from it, and abuse it. We're revealing our security flaws, but are we fixing them? If we're not, people will use those security flaws.

Just my two cents and concern. At any rate, I'd love to see them both worked on more, but I think we should hesitate a little before we just jump to "throw it on github for all to see".
Logged
Offline (Male) polygone
Reply #5 Posted on: May 26, 2013, 03:51:02 pm

Contributor
Location: England
Joined: Mar 2009
Posts: 794

View Profile
I want a way of knowing if somebody makes a comment on a game I posted at the EDC.
Logged
I honestly don't know wtf I'm talking about but hopefully I can muddle my way through.
Offline (Male) Josh @ Dreamland
Reply #6 Posted on: May 26, 2013, 10:58:01 pm

Prince of all Goldfish
Developer
Location: Pittsburgh, PA, USA
Joined: Feb 2008
Posts: 2949

View Profile Email
Polygone[0]: I guess I can see merit in linking 64Digits files, but then we don't have control over whether the user can delete the file without deleting the game. If the EDC controls the files, then users can't accidentally break links.

Forthevin:

Regarding the licensing issue: those were basically my thoughts, though the attribution clause concerns me; GPL does not require attribution except insofar as the license remains intact, and whether that is sufficient to meet the three clauses of that BSD license was unclear to me. Also, that still presents an issue if we want to put the theme up on GitHub, though I don't see how they could limit our ability to do that if their license is GPL compatible.

Regarding the regression tests: I've wanted to make a regression testing suite for a long time, but I never had the wherewithal to get the job done myself. The task is every bit as daunting as documenting all the functions; it'd be an endeavor for the entire team. The most intricate testing suite we ever had was TGMG's compiler test, which, using a collection of a couple thousand examples harvested from 64Digits, compiled each one sequentially with ENIGMA, noting missing functions. I have no idea what has become of this testing environment, but TGMG should be back with us in a month or so.

Regarding your EDC points: I had meant to express that the multiple download links were for each platform, but I forgot that they might also be used for mirrors and assumed that the idea was implied. But yes, that feature is definitely on the radar. I really like the HTML5 embedder idea; I was actually thinking about using TGMG's old Java applet that embeds ENIGMA games (talking of GMBed). Then users could play any ENIGMA game from within their browser, sort of (the game would still be an EXE running on their machine, but it'd give the illusion of running in their browser). I do like the HTML5 idea, though; I forgot to consider it. You can currently edit your own EDC submissions, but the thing is presently devoid of moderation features. So if someone makes an ass of themselves on it, I have to delete them manually using SQL queries, which will probably piss me off enough to IP ban that person. That's definitely also on the radar. What 64Digits (the site upon which the EDC is based) does is use an approval queue for all games; games aren't displayed until a moderator checks that it isn't a virus. Of course, that only works if it's not on a timer, or something, but it's still one decent measure.


IsmAvatar: I hope you're wrong about this, for once. Most of my idea is that security isn't an issue. The reason for this is that all of the SQL queries go through the SMF API, which handles all sanitation for you. Thanks to that, the only kind of exploits anyone will find are generic JavaScript-flavored XSS. The database access is all handled by SMF's library, which would remain far from the GitHub. And yes, most of our manpower is tied up in the engine, and that is right where it should be. But I mentioned the case with Robert as a good example of something versioned site code will fix; now no one needs me to edit theme-related code, because it's all right on GitHub. Meanwhile, the stuff I do *not* want people editing, is *not* on GitHub. Granted, if Robert wastes more time on that, that's still time he could have been spending on the project. But he's going to waste my time, too, if that's not an option. And keep in mind, some people will come along who want to contribute, but don't know any C++. Some of those will be web developers who might want to lend a hand.

Polygone[1]: That can be arranged.
Two other missing features of the EDC:

1) It does not correctly set the currently active page in the forums. It shows up as "Unknown action."
2) There's no way to watch threads/blogs/users/general activity.
Logged
"That is the single most cryptic piece of code I have ever seen." -Master PobbleWobble
"I disapprove of what you say, but I will defend to the death your right to say it." -Evelyn Beatrice Hall, Friends of Voltaire
Offline (Unknown gender) forthevin
Reply #7 Posted on: May 27, 2013, 02:58:30 pm

Contributor
Joined: Jun 2012
Posts: 167

View Profile
In regards to the licenses, I am certain that GPL3 and 3-clause BSD are compatible, since the Free Software Foundation states that they are compatible: http://www.gnu.org/licenses/license-list.html#GPLCompatibleLicenses.
In regards to distribution, the issue is that SMF versions 1.0 and 1.1 are licensed under the Simple Machines license, while SMF 2.0+ are licensed under 3-clause BSD. I looked into the source for SMF 1.1, and their themes are also licensed under the Simple Machines license. Therefore, the themes cannot be redistributed without their permission.
I am not certain whether we can redistribute the SMF 2.0+ software, though it may be allowed, since I don't think using SMF 2.0+ with a 1.x theme counts as a modification or derivative to the 1.x theme.

In regards to security, I think there's a better view of things: I think it is a safe bet that no matter what we do, the website is going to get cracked sooner or later. Open sourcing parts or not will not make much of a difference in regards to security, either positively or negatively, but it will make a considerable positive difference in regards to development and maintenance. Therefore, I think the best idea is to version control the things we want to version control, try our best to make things secure, and otherwise ensure that nothing is lost when the website gets cracked. This means that we shouldn't have any personal information anywhere, that we should discourage people from writing things they don't want in public in PMs, and encourage people to only use passwords on the website that are entirely unlike any of their other passwords, and finally backup anything that we would not like to get lost.
One of the things that convinced me of this was looking into how SMF hashes passwords. In the current version of SMF, SMF 2.0.4, password hasing is done by using the username as the salt, and using SHA-1 for the hashing. This is not a very good way of doing things, as explained on this website: http://crackstation.net/hashing-security.htm. Given that SM does not get that part right, I suspect that there are other parts in SMF that they do not get right either.
Logged
Offline (Female) IsmAvatar
Reply #8 Posted on: May 28, 2013, 10:26:26 am

LateralGM Developer
LGM Developer
Location: Pennsylvania/USA
Joined: Apr 2008
Posts: 877

View Profile Email
Sounds good to me. Just beware before you throw EnigmaBot up on the github, as I know for a fact that he stores his credentials in either a plaintext file or hardcoded, which gets "distributed" right along with him. This information would naturally need to be censored. Otherwise, I'd like to see him AND thundercleese (thevalog) up on version control so that we can begin merging them. Again, censoring where appropriate with thundercleese.
Logged
Offline (Male) Josh @ Dreamland
Reply #9 Posted on: May 28, 2013, 04:40:56 pm

Prince of all Goldfish
Developer
Location: Pittsburgh, PA, USA
Joined: Feb 2008
Posts: 2949

View Profile Email
Thundercleese is already on GitHub, probably with a full SQL password along with Gary's IRC credentials. I'm aware of EnigmaBot's credentials file, and I'm weary of other files with SQL passwords.
Logged
"That is the single most cryptic piece of code I have ever seen." -Master PobbleWobble
"I disapprove of what you say, but I will defend to the death your right to say it." -Evelyn Beatrice Hall, Friends of Voltaire
Pages: 1
  Print