Konstantin has posted 1 posts at DZone. View Full User Profile

DSL Adoption with JetBrains MPS

04.07.2009
| 23957 views |
  • submit to reddit

DSLs, or Domain Specific Languages, have been known in software engineering for many years. Despite this fact, they aren't widely used today. In this article we take a look at what DSLs are and why they aren't widely accepted by mainstream developers. Then we describe how JetBrains MPS solves the main problems which stop DSLs from being widely used.

A DSL is a language tailored to a particular problem domain. For example, declarative languages designed to solve narrow domains of problems are DSLs. Among such languages are SQL, Regular expressions, XPath, and Prolog. The main advantage of DSLs is that they are very close to problem domains, which implies that problems from the domain can be specified and solved very succinctly in such languages. Another advantage is that in order to code in a DSL, a person doesn't have to be a software developer. You can give a DSL to a domain expert, and she can write code in it thanks to her knowledge of the domain itself.  This looks simple in theory: just take your domain, look at its abstractions, create a language for it, and describe/solve your problem in this language. But despite this simplicity, we rarely see DSLs in real world programs.

Why DSLs aren't widely used in mainstream programming

Let's take a look at what prevents DSLs from being widely used. The first reason is focusing on closed DSLs rather than on creation of general-purpose language (GPL) extensions. This is probably because of potential ambiguities in text-based languages. The other problem is the complexity of creating a good IDE support for such languages. Let's take a closer look at these reasons, one by one.

Most of the effort in DSL community is directed toward standalone DSLs, but we can get much more value if we add new constructs to an existing language, for example Java. If we can use language extensions, in a way we use libraries and frameworks, these languages have much greater value to us than if we just create a standalone language and use it for some part of our program. This is especially noticeable when we have languages intended to be edited by professional developers. With this approach developers can have the high abstraction level of DSL and the power of GPL at the same time, which isn't achievable in existing DSL technologies.

In addition to being able to create extensions for existing GPLs, we need to make them composable since reuse is hardly possible without this property. This means that if we have a language extension, say one that adds support for monetary values to Java and another language extension, for example one that adds support for useful mathematical notation to Java, we must be able to use them in the same program simultaneously despite the fact that they are created by different vendors.

In order to solve GPL extensibility and composability problem, we need to first solve another important problem: problem of text-based language ambiguity. Almost all of the existing languages are implemented with text-based grammars. These grammars have one very upsetting issue: they can be ambiguous, i.e. there might be several interpretations of the same program. Even worse, even if we make the grammar of <GPL X + extension A> unambiguous and make the grammar of <GPL X + extension B> unambiguous, this won't mean that the grammar of <GPL X + extension A + extension B> is unambiguous. As a result, such extensions won't be composable, which greatly reduces their value.

In order to be productive, a developer needs intelligent tools at hands. With the invention of intelligent code editors such as IntelliJ IDEA and Eclipse, developers feel really bad when they have to use plain text editors for other languages. These editors don't highlight errors, don't provide context-sensitive help, and don't show completion menus with possible values. There are frameworks for such editors, for example IntelliJ IDEA Language API, XText and Oslo but neither of them supports composable language extension. Even with these frameworks, creating an industrial grade language support requires substantial knowledge of programming languages and takes a lot of time. As you can see, tools support is very important for developers but such tools are very hard to implement.

Let's summarize the main problems:
People focus on a wrong kind of DSLs. In order to achieve substantial productivity gain, we need to create GPL extensions instead of DSLs. This problem arises because text-based languages are not composable and this composability isn't supported by current technologies.

How MPS solves these problems

Now let's take a look at how MPS solves the problems described above. In order to allow language extension, MPS doesn't work with program as a text; instead it stores a program as an abstract syntax tree (AST) and edits it directly. This choice of storage greatly reduces complexity of creating IDE support. Since MPS has the complete syntax tree at hand, it can easily provide context-sensitive completion, error checking and other crucial aspects of an intelligent editor.

MPS solves the problem of text grammar ambiguity in a radical way: by not storing language code as text. If there's no text, there's no text grammar, and thus no need to use a parser with all of its associated grammar ambiguities. Instead, MPS stores each program directly as a syntax tree. This does not mean that languages in MPS don't have grammar. They have one, but it isn't text-based or concrete grammar - it's an abstract grammar. Abstract grammar describes the structure of the program's syntax tree. If you are familiar with XML, MPS's abstract grammar is very similar to XML Schema.

As grammar ambiguities are eliminated, languages can be easily combined with each other.  You can add new constructs to an existing language, which is called language extension. You can embed General Purpose Languages (GPLs) inside of DSLs as well. This means that languages are composable, which promotes reuse of languages. In MPS, we have done a lot of experimenting with language composability. We have created many Java extensions:

  • Collections Language, which adds first class support for collections in the style of C#'s Linq;
  • Dates Language, which allows working with dates more easily;
  • Math Language, which adds mathematical constructs like sums and intervals directly to the language; and many more.

Most of the language definition languages that we have in MPS both embed and extend Java. For example, Type System Definition Language has type system rules which look like usual DSLs, but instead of having custom syntax there, you write Java code within special constructs inside of the rules.

Since we got rid of the text-based storage, we cannot use a regular text editor. In order to edit the syntax tree, we use a special editor. In MPS, we have a projectional editor. For each syntax node, it creates a projection which can be interacted with, and as a result of these interactions, the syntax tree is changed. In MPS, we've done our best to make this editor as close to a text editor as possible. For example, if you type 1+2+3, the syntax tree for this expression will be created. You don't need to choose PlusExpression from the completion menu twice - our editor behaves just like a text editor in many respects. Of course, not everything that is possible in a text editor is possible in the MPS editor, but all such issues can be worked around without losing productivity. In our experience, a person can get used to our editor so that she is as productive there as in a text editor in two weeks' time. 

As a result of syntax-tree-based storage, smart editing features are much easier to implement. In fact, many of them are provided automatically, for example code completion, find usages, and version control support. During the development of IntelliJ IDEA, we have created support for many text-based languages. Implementing such support takes at least several man-months. In MPS, similar features can be implemented in several days. This is
so easy because we have the existing infrastructure for such features and languages with which this infrastructure can be configured.

MPS isn't just an editor. You can create a complete IDE for your languages with its help. We have languages for editor customization, scoping rules definition, type systems, generators, and data flow. All these languages are created with themselves, fully applying the bootstrap principle.

We have also been using MPS for developing commercial software. Our new bug tracking system, code named Charisma, is created completely with MPS, and more programs are on the way.

Conclusion

DSLs aren't widely used in mainstream programming for two main reasons: because text- based languages are usually impossible to combine, and because it is difficult to create an intelligent editor for them.  MPS solves both of these problems by getting rid of text-based code storage and providing a common infrastructure for intelligent editor creation.

MPS Beta 2 was recently released under Apache 2.0 license. We are going to release MPS 1.0 in May, 2009. You are welcome to download it from http://www.jetbrains.com/mps and start creating DSLs right now.

Published at DZone with permission of its author, Konstantin Solomatov.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Arek Stryjski replied on Tue, 2009/04/07 - 11:31am

Could you tell more about version control support. I was thinking all of VCS from CVS to Git expect code to be a plain text. How merging, diff etc. works if you use binary files?

Anyway this is very interesting topic. Specially now because it looks like Java will not evolve very quickly, if at all.

Could this be an alternative for language extension? Maybe to some extend also to new Java languages like Groovy, Scala or Fan?
It will give developers complete freedom in extending Java, but it will lock them to single (and non free) IDE. But maybe it will be worth to do it? And maybe it will be not a case if it is going to be released under Apache 2.0 license.
Still giving up source code (as we know it) feels dangerous...

Could you also comment on this please.

Konstantin Solomatov replied on Tue, 2009/04/07 - 12:15pm in response to: Arek Stryjski

Hello, Arek, We have merge tool which works with any VCS supported by IntelliJ IDEA. It handles merge when you updates and presents model files so that you can merge them in a way close to plain text. I didn't undestrand your second question, could please reformulate it? Source code is available now, you can just download it and change in a way you want.

Alessandro Santini replied on Wed, 2009/04/08 - 9:12am

Privet Konstantin,

I am looking at the DSLs adoption problem from another perspective - resources skills.

Put yourself in the shoes of a consulting company taking over the maintenance of an already existing software application. Assume that this application leverages DSLs written using MPS.

Now multiply this situation by 10 different customers, all of them with different DSLs (even if in the same industry) and now let's get to my questions:

  • How many DSLs must the average team member learn?
  • What is the timescale and financial impacts of learning these languages at every customer engagement?
  • Are these DSLs functionally complete? Are they well documented?
  • How can I access already-skilled resources on the market when talking about non-standard (well, not anymore) languages?

Although I appreciate the benefits DSLs introduce, I can also foresee a lot of risks for which I do not see a simple mitigation. What's your take at this?

Thanks,

Alessandro

Jacek Furmankiewicz replied on Wed, 2009/04/08 - 2:41pm in response to: Alessandro Santini

Think about it as an opportunity :-)

How much more can you charge the client because you have to learn and maintain their proprietary, custom DSLs that no one else in the world knows.

That's gotta be worth at least double the regular rate (you need an Architect, or maybe two of them, etc, etc.).

IT Complexity is the backbone of IT consulting, no? *grin*

Konstantin Solomatov replied on Wed, 2009/04/08 - 3:20pm

Alessandro, DSLs or language extensions which I am talking about aren't new languages. They extend existing languages in the same sence as Java libraries extend Java. Most of these libraries have idioms, best practices and other language like constructs. For example, most of Java developers know that if they work with InputStream they should write something like this:
try { } catch (IOException e) { ... } finally { try { is.close(); } catch (IOException e) { } }
It's the best practice of working with I/O stuff but we can use language extensions to write:
with (is) { .... }
instead.

Alessandro Santini replied on Wed, 2009/04/08 - 5:03pm in response to: Konstantin Solomatov

Kostantin,

thanks for the clarification - the concept was clear to me but still my questions are unanswered: having such little DSLs can bring to a universe of succint syntaxes that are "extending" the base language; however, the case you are presenting is (for clarity and for marketing reasons) the simpliest.

In Italy a proverb says "hunger comes eating" - and that's where my concerns start - when you like the food and you make it more complex.

 Just a footnote: your example looks quite more like a macro than a micro-DSL.

Alessandro Santini replied on Thu, 2009/04/09 - 4:42am

I would totally agree with you if we were not in such tough economic times... :)

Pankaj Shukla replied on Thu, 2010/01/21 - 9:28am

Bankruptcy - Debt Mediators help you with debt problems through debt consolidation with debt agreements and advice on bankruptcy.

Alex White replied on Sun, 2014/09/21 - 9:53pm

Here is one more great resource about debt consolidation   which can help to get rid out of debt faster and  help with debt agreements and professional advice on bankruptcy

Alex White replied on Thu, 2014/04/17 - 6:38pm

Australian website about life insurance quotes .If you need to get your life insurance quote, make sure you visit it. 

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.