Recently one of my friends on Wave started voicing questions regarding the vast number of different programming languages we have. He had trouble (until the rest of us set him straight) understanding why people go so far to make all kinds of new programming languages and, even more vexing, why people even bother using them. He's OK now, but still it seems like something interesting to write about.

We make new languages, new frameworks, new libraries, new runtimes, new environments, new whatevers in order to reduce the immediate complexity of a given task. That is, in order to make various common tasks easier or, in some cases, completely invisible to the developers trying to solve higher-level tasks. Of course, the task's complexity is absolute and isn't really reduced; it's just that the apparent complexity of the task has been reduced.

For any given task it's going to take a certain minimum number of CPU cycles to complete, regardless of what language it was created in. This is the actual or absolute complexity of the task. What languages and frameworks do is hide away much of this complexity using automation and encapsulation so that programmers don't have to worry about those details. Therefore, their minds are freed to concentrate on the task and so it seems a lot less complicated than it really is.

General languages have a number of priorities that are understood by its creator(s). One language may treat interoperability as its most important feature, another might be ease of use, or simple speed. For this type of language, new languages are created to fill gaps in these priorities: if there is no object-oriented language where ease of use is the most important feature, then someone will step in and create one. When you need an easy-to-use object-oriented language, you should naturally be inclined to use languages that satisfy that role. The apparent complexity of applications written in general languages is moderate.

The other major branch of languages is domain-specific languages. These are languages or frameworks that are specifically built to satisfy the needs of a very small domain of problems. Unlike general languages which have general priorities, domain-specific languages' priorities are specific to the problem area. These languages pre-solve most of the problems in the domain, so if you find yourself writing software in that area you find that you don't have to do much work at all. The apparent complexity of domain tasks in the right domain-specific framework is therefore miniscule.

Some examples of general languages include Java and C# (as well as countless others — I don't presume to have an exhaustive list), both of which have some priorities in common and many differences as well. Both of these languages have immense frameworks dedicated to reducing the complexity of many tasks by handling a lot of the details for the programmer, such as memory management. However, their frameworks do not go out of the way to solve specific problems, but rather provide the tools for solving a wide variety of problems on equal ground. SQL and regular expressions are domain-specific languages: SQL for the domain of relational database interaction and regular expressions for deciding regular languages. In those domains, it is pretty silly to use other languages because, almost universally, those tasks will have higher apparent complexities in other languages.

The problem with domain-specific environments is when you need to do some peripheral task in addition to the domain task the language is designed to solve. The drawback of domain-specific frameworks is the lack of perfect foresight of its designers. The designers see the problems in a certain way, and went about solving all of the problems they were able to see from that point of view. But you have a different point of view and can therefore see a slightly different set of problems. If you're lucky, your problems were anticipated by the designers. If you're unlucky, you're going to be faced with a problem your tool wasn't designed to solve. So you have to either work another tool entirely into your solution, or you need to kludge a way for your tool to work in a crazy, wrong way that somehow, sorta works.

This is more obvious with some languages than with others. For example, trying to parse a non-regular language with regular expressions is impossible. Others aren't so blatant, such as trying to parse strings using XSLT, or trying to come up with a unique design for a blog within the constraints of a blog templating engine (remember, websites are software, so templating engines count as frameworks for the purposes of this discussion). In both cases it can be done — it's just indirect and requires more effort than it would have taken to do the same task using a general environment instead of a domain-specific one.

Worse still is when your framework is built assuming you will take a certain path to find your solution. This kind of overcomplicated tool can be used only within the domain it was designed for, and even then only if the intended solution matches the anticipated solution within the framework. (As an aside, this is precisely the reason many developers find the way SQL handles nulls to be so strange — because SQL is designed with a particular interpretation of null in mind instead of leaving it up to the developer, but that's another discussion entirely.) To quote William Gibson's book All Tomorrow's Parties:

The handles of a craftsman's tools bespeak an absolute simplicity, the plainest forms affording the greatest range of possibilities for the user's hand.

That which is overdesigned, too highly specific, anticipates outcome; the anticipation of outcome guarantees, if not failure, the absence of grace.

When you're using a tool that anticipates a certain outcome, then using that tool for that outcome trivializes the apparent complexity of that outcome, but doesn't let you apply your own finesse to the result. Additionally it raises the apparent complexity of every other peripheral task. It's important to recognize what your tools are designed to accomplish, so that you don't use the wrong tool for the job. It's equally important to recognize what other tools exist so that you can use the most appropriate combination of tools to reduce the apparent complexity of your effort.