Mr. Lott has been involved in over 70 software development projects in a career that spans 30 years. He has worked in the capacity of internet strategist, software architect, project leader, DBA, programmer. Since 1993 he has been focused on data warehousing and the associated e-business architectures that make the right data available to the right people to support their business decision-making. Steven is a DZone MVB and is not an employee of DZone and has posted 139 posts at DZone. You can read more from them at their website. View Full User Profile

End User Programming -- Solution or Nuisance?

08.08.2010
| 5796 views |
  • submit to reddit
The question of "customization" and "extension" is a common one. For example, "non-technical users to be able to extend with simple scripts".

Read this question on Stack Overflow: "Programming / scripting language aimed at non-technical people".

This is -- at best -- little more than an attractive nuisance. At worst, of course, it turns into a maintenance nightmare and the reason for killing the project. Let's look at the ways this can end badly.

1. Your "end users" can't or don't customize it with small scripts. Bottom line: you wrote a scripting interface which only you can use. Not all bad, but hardly worth the effort. Here's one version of rule: "Make it possible for programmers to write in English and you will find that programmers cannot write in English."

2. Your "end users" turn into the worst kind of hackers and break things left, right and center. Bottom line: you spend more time debugging the messes created by scripting.

3. The real world use cases turn out to be more complex than you bargained for. You spend endless time extending and modifying the scripting support to try and balance limited power (to prevent problems) against enough sophistication (to handle unforeseen use cases.)

"Well," you argue, "those are extreme cases. There has to be a middle road where the user's behave properly." No, not really. This middle road doesn't exist. Why? Programming is hard. A phrase like "simple scripts" is contradictory.


War Story I -- The SAS Hacks

At one organization, the accountants had created a monster. Their chart of accounts in the General Ledger (GL) was not consistent across all of the organizations. Consequently, financial analysis programs required careful rearrangement of certain ledger totals to make them reflect the business reality, not the legacy GL mistakes.

So they wrote some "simple" business rules in SAS do GL extracts. The extracts which were processed by these rules were useful for financial reporting because the legacy GL work-arounds were properly allocated.

Our job was to write a proper financial reporting tool that correctly extracted and reported on ledger data. It was going to be a proper data warehouse. The facts were amounts, the various dimensions were time, account, organization, bank product line, etc.

However, because of the SAS hacks, we could not get a single straight answer to any question. Everything had to be explained by a "here's some code" answer. Their business rules numbered over 9,000 individual simple statements.

Question: How is 9,000 of anything "simple"? Answer: it isn't. The volume makes it complex.

Further -- of course -- the business rules were actually more complex than claimed. The superficial claim was that the rules implemented line-of-business (separate from organization) rules. Some statistics showed that the rules likely contained two independent dimensions of each amount.

The "simple" rules were used to create huge complexity. It's human nature. The world is complex; we want to write software which reflects the world as we encounter it.

War Story II -- The MS Access Hack

At another organization, we had helped the end users buy really nice reporting tools from Cognos (before they became part of IBM) and Business Objects (before they became part of SAP). The tools were -- at the time -- the state of the art in flexible end-user reporting. Cool drag-and-drop interfaces, flexible scripting add-on capabilities. Really sweet.

Did the users actually use this power?

No.

They wrote dumb, large, detail-oriented reports. They extracted the bulk data into MS-Access Databases. Then they hacked around in MS-Access to produce the reports they wanted.

Why? (1) They understood Access; learning to use Cognos or BO tools wasn't going to happen. (2) They wanted real power; the limited scripting capabilities weren't solving their problems.

The reason we became involved was because their Chief Access Hack (CAH™) got another job and no one could figure out how the Access stuff worked.

After two weeks of reverse engineering, I found a bunch of business rules that should have been in the data warehouse. We had a nice meeting with their executive; we wanted to talk about the CAH and what they were going to do to prevent this kind of MS-Access nightmare in the future.

The executive reminded us that business rules are subtle and complex. Data warehouse software development is slow and clumsy. The users will always be several steps ahead of the warehouse processing. The MS-Access hacking is how they coped with the limitations of the warehouse.

Bottom Lines

Software is complex. We can't create a language suitable for "non-technical users". Either the language is too technical for our users -- and it isn't used -- or the users become technical and demand more features.

People learn. They're only non-technical users on day 1. If they start using the scripting language, they immediately start on the path to become technical users.

The real world is complex. There are no "simple scripts" except in a few textbook cases. When we first set out to build our simple scripting environment, we start with a few sensible use cases. Then unforeseen use cases crop up and we either tell people they can't use our system [Really, would you turn them down?] or we have to add complexity.

Architectural Implications

Because of this complexity issue, avoid some urges to refactor. Not all refactoring is good refactoring. There is A Limit to Reuse. Specifically, any attempt to create plug-in scripting into a larger application rapidly becomes hard to manage. It's better to provide necessary libraries to build applications around a script than to plug a script into a standardized application.
References
Published at DZone with permission of Steven Lott, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Chris Treber replied on Tue, 2010/08/10 - 5:37am

First one, than the other.

Many companies seem to be running on Excel sheets, which first is a great tool to get stuff done yourself, and then, sometimes, some sheets become business critical applications with an untangable mass of macros that nobody understands, that can't be maintained or extended, or - that would be the best - that can be replaced with a decent solution

Kristian Rink replied on Thu, 2010/08/12 - 3:51pm

Your points are valid and your examples are, too. There's only one problem I see here, talking about languages for "non technical" people: Do these "non-technical" people _really_ want to have _all_ the features an up-to-date scripting languages (take Python as an example) has to offer? Including, eventually, technical things like sockets, threading, client/server programming, or "esoteric" things like closures, lambda expressions and the like? Or would they rather prefer a straightforward, powerful language which is as good as somewhat possible in sync with the terminologies, concepts, ... of their (business/problem) domain rather than being a "technical" general purpose language with a general purpose, leaving them think about way too many things which still might be too technical for them?

 

K.

King Sam replied on Fri, 2012/02/24 - 10:51am

There is a third way. At one client, the "power users" did their thing until the report became "stable". "Stable" was a negotiated definition on a per report basis between the "power user" and management. Once it was agreed that the report was "stable", it coded up into the formal infrastructure. Yes, there were still politics. The "power user" did not want to give up his baby. Management had to say, go have "another baby".

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.