Tips for making an IDE?

[Mangolion]

I am currently working on a small but very radical IDE for my custom language. What I'm trying to do is to create an object oriented language for note taking in classrooms (or note taking in general). For example, in Geography class, I can make an object called Sphere and then create sub objects called Lithosphere, biosphere...
I already have the object language syntax marking done, along with code suggestion, I can also reference these objects in normal typing. Oh and since this is only a scripting language, space between variables are permitted
Here is a simple screenshot:



While the development of this IDE is going smoothly, I still haven't got to the hard part yet (implementing images, sounds and camera for taking notes ...)
If you look close enough, you'll see small white rectangles, that would be my attempt to give it a moving background, like the live wallpaper on Androids. Its based on a dynamic background library that i wrote which is basically a small particle system. I wanted to style my IDE with themes such as falling snow when its Christmas, flying bats on Halloween... I have a lot more themes too (aquarium, sky with birds and clouds, ...). I try to make the background dark and move slowly so as to not distract the user.
What I am asking is, when you guys make an IDE, what were the hardest things that you have encountered, and how did you solve them?

Here are some links if you wanna take a look, I use Eclipse.
Executable Jar: https://www.dropbox.com/s/gpp41kfjbsa6xuv/Ellowyn.jar?dl=0
Source: https://www.dropbox.com/s/etih84ozb13r577/Ellowyn%2010-22%2011-45pm.zip?dl=0

[lonewolff]

Nice one! 

I am about to start doing a similar thing with my game engine also.

What are you using to create the IDE?

[Goombert]

Very nice work Mongolian! I like to see all this innovation indie developers are trying to put out there, there's a lot of interest in the indie community today to start innovating and break old barriers of software and game design. I am really impressed at the initiative people have been taking. I really like the schematic rendering you do, Josh tends to do a lot of that in JDI for testing and to properly outline the architecture of the compiler. I have always been a fan of entity-component systems like Unity3D, I think they are the epitome of object oriented programming. I already like your scripting language because it is similar to QML from the Qt Framework, it breaks a lot of the barriers imposed by C++ while being able to share objects between code in both languages.

[IsmAvatar]

Founder of LateralGM here. Thought I'd give my two cents on my methodology that helped get LateralGM to where it is today.

I like to run things by learning from the mistakes/failures/shortcomings of others. Game Maker doesn't work on Linux? I'll make mine work on Linux. G-Java evaporated from massive feature creep? I'll avoid features and focus on core functionality. This methodology reflects in all of my projects.

That said, one of the most common mistakes I have seen here in the ENIGMA umbrella community (at least, in the years when I was mostly active) is starting a project with features (massive feature creep from the beginning). Not in ENIGMA itself, but in side projects, and especially IDEs. Don't get me wrong, every project requires an idea - something new - a feature, but many projects here get tied up in features and leave the core functionality and bugs to rot in the gutter. And that's the last you hear of those projects - they evaporate in a brilliant burst of colorful features with no substance.

As such, I always advise people starting new projects to identify a core set of features, the "idea", and build the functionality for it. Focus on the core - the meat and bones - the part that actually gets stuff done - function over fashion. There will be plenty of time to add pretty bells and whistles later, AFTER the core functionality is done.
As you're developing the core functionality, you'll think up a bunch of new features that would be neat to add to the project - features that you probably wouldn't have thought of if you hadn't focused on the core. That's great, write them down in some central place with all the rest of you feature ideas. You can work on them later, when you're actually doing features. Keep them in mind while you're building your project so you know to build certain things more modular than they have to be so that those features are easier to implement later.

As for each idea, feature, etc (the language, the file format), do take some time to flesh it out, and find someone else to bounce ideas off of. If you keep revisiting some piece of code after the fact to add something you missed, the code will quickly start to deteriorate into something ugly and unmanageable. Refactoring is always an option, but code that's been touched up a lot of times tends to have its ugly greasy fingers in everything, making refactoring really difficult because you find yourself breaking a lot of other things. Think it out first, define the core set of functionality, and then implement it. You'll be very pleased with the result, and with a bit of luck, you'll never have to touch it again (until someone decides they want to change the code formatting...).


That's my advice. The world needs more people who take the time to do it right.

[Mangolion]

Thank you, IsmAvatar, you have told me exactly what I need to hear. I admit that I have been pushing the project too fast, my trial and error parser is now practically unreadable, I'll have to rewrite it soon.

I wanted to use this project in school so badly that I have been working day and night on it, trying to add cool features without really thinking of its long term usability. I will now slow down and document my code better. However, I'm still going to implement images, sounds and videos into these notes soon! I would be really happy if you guys could download this IDE and try it (not know, its not complete) and give me feed back 

Lonewolf, I am using java Swing to make the IDE, Swing seems to be a good and easy option. What kind of game engine are you working on and may I see it?

[lonewolff]

Founder of LateralGM here. Thought I'd give my two cents on my methodology that helped get LateralGM to where it is today.

I like to run things by learning from the mistakes/failures/shortcomings of others. Game Maker doesn't work on Linux? I'll make mine work on Linux. G-Java evaporated from massive feature creep? I'll avoid features and focus on core functionality. This methodology reflects in all of my projects.

That said, one of the most common mistakes I have seen here in the ENIGMA umbrella community (at least, in the years when I was mostly active) is starting a project with features (massive feature creep from the beginning). Not in ENIGMA itself, but in side projects, and especially IDEs. Don't get me wrong, every project requires an idea - something new - a feature, but many projects here get tied up in features and leave the core functionality and bugs to rot in the gutter. And that's the last you hear of those projects - they evaporate in a brilliant burst of colorful features with no substance.

As such, I always advise people starting new projects to identify a core set of features, the "idea", and build the functionality for it. Focus on the core - the meat and bones - the part that actually gets stuff done - function over fashion. There will be plenty of time to add pretty bells and whistles later, AFTER the core functionality is done.
As you're developing the core functionality, you'll think up a bunch of new features that would be neat to add to the project - features that you probably wouldn't have thought of if you hadn't focused on the core. That's great, write them down in some central place with all the rest of you feature ideas. You can work on them later, when you're actually doing features. Keep them in mind while you're building your project so you know to build certain things more modular than they have to be so that those features are easier to implement later.

As for each idea, feature, etc (the language, the file format), do take some time to flesh it out, and find someone else to bounce ideas off of. If you keep revisiting some piece of code after the fact to add something you missed, the code will quickly start to deteriorate into something ugly and unmanageable. Refactoring is always an option, but code that's been touched up a lot of times tends to have its ugly greasy fingers in everything, making refactoring really difficult because you find yourself breaking a lot of other things. Think it out first, define the core set of functionality, and then implement it. You'll be very pleased with the result, and with a bit of luck, you'll never have to touch it again (until someone decides they want to change the code formatting...).


That's my advice. The world needs more people who take the time to do it right.

1000000% agree with you. That is me right there down to the core.

Lonewolf, I am using java Swing to make the IDE, Swing seems to be a good and easy option. What kind of game engine are you working on and may I see it?

I have actually started a thread about it.

http://enigma-dev.org/forums/index.php?topic=2313.0

Starting simple and solid from the ground up.

[DaSpirit]

I like to run things by learning from the mistakes/failures/shortcomings of others. Game Maker doesn't work on Linux? I'll make mine work on Linux. G-Java evaporated from massive feature creep? I'll avoid features and focus on core functionality. This methodology reflects in all of my projects.

This is one thing I agree with. Having failed to make an IDE (and thinking about getting back on the horse every once in a while), I know that you really have to plan everything out.

I've been planning a lot myself recently (but not features, only the core). You have to think about how you want everything to connect to each other. What I mean, is that if your IDE can't do something else that another IDE can do, then it has a flaw. If you make your core API correctly, then you can add anything new with it.

Josh has injected the idea of plugins into me. If you make a plugin API and do it correctly, then you're technically done with the IDE, because everything else can be a plugin, separate from the actual IDE. Your IDE just tells everything how to interact with each other. If you were to figure out parts of your API later, you may have to rewrite large portions of your code to get the results you want, wasting a lot of your time.

[Mangolion]

How does one write a plugin API in general? I haven't much experience in writing (anything) actually. I've only started slowly learning Java for the last 2 years. Should I write this "API" by having other developers write their custom classes and then I'll use a classLoader to load their classes on run time, like Minecraft Forge (you'll know if you write minecraft mods)?
How was the enigma plugin for lgm written?

[DaSpirit]

I don't know about LGM, but I'll share my notes here. My API resolves around plugins being able to inherit from five classes. I mention "SerialData" often, which is an abstract data type. Think of XML, when you think of this, but in memory. It consists of a node that may or may not store children child and may or may not contain its own data. The idea is that you convert data formats into this one unified format (so you convert XML, JSON, YAML, or even GMK into SerialData) and then you can convert any SerialData back (so you can switch file formats - the file format won't even matter internally).

Here are my current notes (I think it may be complete, but I haven't put anything into practice yet):

Code: [Select]

Natural Game Maker (NGM)
=========================
API Design
----------

NGM is built up on plugins. Plugins may add Editors, Compilers, Projects
(Serializers) and Actions.

Plugins
 - External manifest (useful to know which plugin failed, if one fails).
 - Has a load order (for dependencies).
   - Dependencies can be optional.
   - You can't unload a plugin that has dependencies active.
   - Plugins don't know about plugins they don't depend on.
 - Plugins may store callbacks.
   - May connect to application callbacks (e.g. random new project opened).

Projects load resources.
 - It turns these resources into SerialData.
 - SerialData is used to pass other parameters around.
 - There can be multiple projects of the same project category.
 - These serializers are stored in whatever resource they create (for safe
   modification).
 - A project can contain a group of resources or a standalone resource (e,g,
   text file).
 - Importing and export of projects is also done by serializers.
   - You can import/export file types with a regular save for standalone project
     resources.
 - Projects may also optional set compiler configurations.
 - Other plugins may connect to project callbacks.
   - Plugins can only acccess mutable project SerialData if they have the project's plugin as a dependency.

Compilers compile resources.
 - Compiler configurations are added to resources.
 - Configuration settings are done via SerialData.
 - Compilers are mapped to project categories (even sub-categories).
   - Sub-category groupings allow the compiler to work for a range of projects.
 - When running in debug mode, the compiler should have the project structure.
   - It can open editors for resources.
     - It should be able to add callbacks to the editor (e.g. show crash
       location).
     - It should also be able to add editor callbacks into itself (e.g. step
       over).

Editors edit resources.
 - Editors create GUI elements from the resource SerialData.
 - Editors may store callbacks.
   - Requirement for compilers.
 - Has to connect to the actions on its current context (e.g. window tool bar).
 - Can have cloned editors (multiple editors for single resource).

Actions for misc. functionality.
 - Executes a function.
   - Can access everything from whatever the plugin uses.
 - Can be stored in a standard location (e.g. file, in the menu bar).Copyright (c) 2014 Daniel Hrabovcak

Feel free to use any of my ideas. If you do use it, you don't have to give credit. A lot of these ideas were suggested by Josh (the ENIGMA project leader).

The biggest part of writing an API is deciding everything you need, and how much you want to give users access to (and also what you don't want to give users access to).

[IsmAvatar]

LGM wasn't built with Plugins in mind. When I started the project, I had no experience with plugin APIs, nor did I have any clue how I would apply it even if I did. Once it started to get to the point where I wanted plugins to be possible, I started refactoring some of the core parts, and adding hooks. That said, the plugin-ability of LGM is limited, and a bit of it has had to be refactored every now and then to allow plugins to do more things as they crop up.

So while there's a lot to be said about plugin-ability, it's not completely necessary. It might save you some strife later trying to implement plugins, but I'd say that the refactoring I did for LateralGM at that time was quite enjoyable.

Also, as far as thinking everything through goes - requirements change as you develop. You can't expect to lay out the entire blueprints up front and then just code monkey away at it. Lay out the groundwork first, develop it, and then figure out what parts you need next. LateralGM started with just a non-functional menu and a non-functional resource tree. That was it. That was the first prototype, and it was quite popular. Each part I added to it after that I made sure to put a lot of thought into and fully thought out the structures that I would need to do it. That first resource tree wasn't just a non-functional resource tree. It was a fully structured resource tree, with a data model behind it that actually reflected the state of the game that was in memory. The second prototype simply hooked in the code to load a game (.gm6) into that pre-existing memory structure. And the tree immediately reflected it. No fancy bells and whistles, just core functionality.

[Goombert]

LGM wasn't built with Plugins in mind. When I started the project, I had no experience with plugin APIs, nor did I have any clue how I would apply it even if I did. Once it started to get to the point where I wanted plugins to be possible, I started refactoring some of the core parts, and adding hooks. That said, the plugin-ability of LGM is limited, and a bit of it has had to be refactored every now and then to allow plugins to do more things as they crop up.

I can personally attest to this with the increasing necessity for a revamped plugin system with Project Nashorn support. Also specific things like lookandfeels, in the current beta release of LGM all jars under the lookandfeels folder are loaded into the classpath regardless of whether the look and feel is actively in use. Some special consideration needs to be given to stop the unchecked overgrowth of the current infrastructure with a more evolved system put in place.