Last week, the Microsoft TechEd North America 2010 took place in the great city of New Orleans. I was lucky to be invited to do a talk on how frameworks can kill your projects.

When it comes to Microsoft .NET-connected development, more and more frameworks enter the market. Both from Microsoft and from open source. Think of ASP.NET MVC, Castle, Windows Workflow Foundation (WF), Entity Framework, Unity, Linq2SQL, ADO.NET Data Services, Windows Communication Foundation (WCF), nHibernate, Spring.NET, CSLA, NUnit, Enterprise Library or ADF.

Trouble begins

Once a project chooses to apply one or more frameworks, trouble begins. What if you require features that aren’t implemented in the framework? What if you decide that another framework would have been better and want to switch halfway through your project? What if the author of your favorite open source framework suddenly stops developing? What if the framework contains bugs or omissions? And, what if a new version of the framework is released that is implemented differently? These and many more everyday problems will cause your project to come to a halt, or at least make you perform serious refactoring.

Demos and (bad) code

During this highly interactive talk, Sander Hoogendoorn, chief architect of Capgemini’s agile Accelerated Delivery Platform, and member of Microsoft’s Partner Advisory Council .NET, demonstrates pragmatic architectures and patterns that will help your projects to stay away from framework issues, and how to keep code independent of framework choices. Sander presents models of layered architectures, and applying bridge patterns, managers-providers, dependency injection, descriptors, and layer super-types.

Of course, Sander illustrates these insightful patterns with lots of demos and (bad) code examples using blocks from Microsoft’s Enterprise Library, NHibernate, Log4Net, and the Entity Framework. Learn how to improve the structure and quality of your software architecture and code, and how to avoid the pitfalls of applying frameworks to .NET software development.

Every now and then you see really horrible examples of software development.

While I was try to scan a document using my printer/scanner, the software by HP failed miserable. Being the nice customer I am, when HP invited me to fill in an enquiry about the quality of their service. The form presented to fill in my feedback was simply horrifying.

Just have a look at the following example from that form.

Please don’t make changes to this field? They must be joking. Am I taped on Candid Camera here?

It’s a mildly sunny April morning in 2002 when I park my car outside of a huge government agency office in a small suburban city near Utrecht. I am invited for a brainstorm session with the agency’s enterprise architects. Although I do not consider myself an enterprise architect, and explained that upfront, they were eager to discuss their architecture with me. Ok.

After registering at the reception, I enter the meeting room. Now this might be specific for the Netherlands, but the enterprise architects are sitting at an oval table, all equal in our consensus world. “So what is it you’re doing?” I ask, while looking at the architect who looks more equal than the others. Happy to be addressed the architect, bearded and all, stand up and walks to the whiteboard, picks up a marker and starts telling their story.

Rectangles and arrows

While talking our bearded architect starts drawing the different application and systems in their system landscape. Each represented by a rectangle with a three to five letter acronym on the white board. The big challenge for this government agency, so I understand, is to from a tightly integrated system landscape to a service oriented landscape with an enterprise service bus.

Next my bearded friend starts to discuss the connections between the different systems and the enterprise service bus. Here come the arrows! “So we connect each system to the bus, and we will have upgraded our landscape for the future,” the architect smilingly concludes. I nod my head willingly. “Looks good,” I start complimenting the architect. “But,” I continue, “how are these arrows implemented?” “Well, that’s easy,” says the architect. Without saying a word he picks up the marker again and slowly adds the acronym SOAP to one of the arrows on the white board. “That’s all.”

SOAP

Now, from an enterprise architect’s point of view that might be all, but from a developer perspective drama begins here. I roughly estimate that each of these fancy little SOAP arrows likely represents about 4 to 8 weeks of work – note: it is still 2002. And there are quiet a few of these arrows on the whiteboard; and the view is likely not the complete picture.

Concluding: what might seem insignificant and trivial from an architectural perspective, might be complicated and elaborate from a developer’s and tester’s point of view. Or as Scott Ambler so eloquently puts it: everything works on a PowerPoint slide. And sorry dear architects, it just doesn’t, no matter how brightly colored and great looking your PowerPoint presentation are.

Wouldn’t it be good if enterprise architects actually participate in projects to see in real life what these simple drawn decisions. Actually, it is good. For some years now, in the agile projects I am coaching enterprise architects, business and information analysts take part during the actual iterations. Instead of upfront, untested architectural demands or long review periods afterwards, they actually participate in the design workshops of the project. And you know what? They love it. It’s simply great to actually see directly what comes out of what you so cleverly think of. And even better: you, as an architect are totally rid of these long and cumbersome review period, and you directly get to influence the way the software is built. Please do.

The penultimate goal

Can you do even better? Well, there’s a penultimate goal. In September 2003 I did a talk at a large software development conference in Denmark. During the speaker’s dinner I found myself at a table with some great names in this field – think of Bjarne Stroustrup, Jos Warmer, Kevlin Henney – and even bigger glasses of cool Danish beer. Nerds as we were, by the end of the evening we concluded that this particular conference should have a panel discussion with 42 panel members, and we would name it The Panel at the End of the Universe. Great thinking!

And about an hour and many glasses of beer later, we agreed upon an even bigger contribution to the field of software development. We thought that the penultimate solution to developing software is to be able to generate software directly from the rectangles and arrows in the architect’s PowerPoint presentations. That would sure boost productivity!  Then we could actually say that everything does work on our PowerPoint slides.

Suppose you are in the IT department of a (very) large organization and have been developing systems for your organization for quite some years. Chances are that you will have a landscape of systems great and small that all serve a particular purpose, or that have served a particular purpose. Systems that were built in a variety of languages, such as Visual Basic, Cobol, C, PowerBuilder, Java, or packaged systems including SAP, Oracle, PeopleSoft or more modern variations such as salesforce.com.

Since each of your systems only covers part of your business processes it is likely that they are linked together in all kinds of ways. Either they re-use each others functionality via services, or worse they use each others data or databases. Or maybe they even just exchange files in XML, Excel, CSV. No matter the protocol, life will be harsh on you.

Release calendar

A few weeks ago I had a really interesting conversation with the release manager of such an organization. He was desperately trying to implement a release calendar with regular intervals. That is, he hoped to reach the situation that this organization had four releases per year of the whole landscape. Each release should implement the results from projects and from handling change requests to existing systems. A noble goal.

However, this noble goal did not seem very realistic. Most change requests on his list included changes to a number of systems. Sometimes up to eight different systems need to be changed to implement a single change request.

The big issue here is that these changes to different systems are aligned in chains. Even making the simplest change to a business process requires development, but much more, it requires testing. Lots and lots of it. First of all the changes to the individual systems need to be tested. Next the chain of changes needs to be tested in the integration test. Then if everything works out fine, the whole landscape needs to be placed into acceptance to perform an acceptance test.

Whiteboard

Standing in front of a whiteboard the release manager drew the different stages in his release calendar on a line representing the three month period he had in mind. “At the end we have the acceptance test,” he stated, and took four weeks from the end of his line. “Before that we run the integration test,” he continued, and took off another three weeks. “And of course the individual systems are also tested, that takes about two weeks after the developers finish writing the software.” And then he finalized his calendar with the statement that business and information analysis also require about two week. “So, there you have it,” he proudly pointed to the timeline.

I stared at the timeline for a while, a bit in doubt whether this was serious, or if  he was pulling my leg. All in all, the release calendar in front of me on the whiteboard only has two weeks to do design and development. Extrapolated that adds up to around eight weeks of software development in a whole year. And this was not even real life, but the situation the release manager strived for to reach in a couple of years. Real life was worse.

We’ve finally done it

And it will get even worse. Because with each change in any of the systems the landscape will get bumpier. Just to give an idea of the complexity, the about forty systems in the landscape are documented in over 9.000 documents, spreadsheets, models, PowerPoint presentations that are located in over 2.000 directories. And with each new bump in the road, testing will get more complicated. Resulting in longer time frames and thus resulting in an even shorter development period. My guess was that within three years from now development will have totally stopped, and the organizations will only do testing. How’s that for productivity.

This is what I call Death by Landscape.

A particularly nasty anti-pattern, this Death by Landscape. It’s consequences not only sink your ability to implement changes and new functionality, it will disallow you to support the new products your marketing department wants to introduce, and it stops you from implementing new legislation or regulations when required. In short: death by landscape directly impacts your business. We’ve finally done it: 

This post was originally published in .NET Magazine. I re-posted it because of the talk I did at Microsoft’s DevDays 2010 in Den Haag recently.  The slides for this talk can be downloaded here.

As you’re probably have been made aware of in abundance, in .Net 3.5 Microsoft introduced a little language feature called LINQ. Although LINQ has been demonstrated at all major and minor conferences, the way LINQ executes queries on already existing types remained slightly less apparent. To this means LINQ introduces a number of new query operators, using yet another new language  feature called an extension method.

Download slides from my DevDays 2010 talk on extension methods here.

Basically, there are two sets of LINQ query operators. One set that operates on objects of type IEnumerable and another set that operates on objects of type IQueryable. To implement the methods in these sets, the .NET architects chose to add something called extension methods, and write the required methods as extension methods to the Enumerable and Queryable classes, instead of directly extending these classes – which would seriously pollute the .NET framework.

Defining a simple extension method

Apart from the apparent use of these extension methods in the query operators of LINQ, there is slightly more to it. At first sight, extension methods appear to allow you to extend any class in any namespace with additional functionality. Let’s have a closer look at them by using an example.

One of things I always disliked about working with the DataSet class is that to obtain the value from a column in a row in a table, I’d have to write code that looks at lot like this:

public void DoTraditionalTest()

{

    DataSet ds = new DataSet();

    object o = ds.Tables[2].Rows[4]["Name"];

}

This code is both tedious and poor code however, as it doesn’t do any check on whether the table and row are actually available. As you’ve probably all have written code like this before, you would probably have appreciated a method like Get() on the DataSet class, like I’m using in the following code example, where Get() performs all necessary checks and formatting.

public void DoTest()

{

    DataSet ds = new DataSet();

    object o = ds.Get(0, 0, "Name");

}

The issue here is that you cannot extend the .NET framework class DataSet to encapsulate this functionality. This is where extension methods come in neatly. With .NET 3.5 I can now write a method such as the following.

namespace Extensions

{

    public static class DataSetExtensions

    {

        public static object Get(this DataSet ds, int table, int row, string column)

        {

            if (table >=  0 && table <= ds.Tables.Count)

            {

                if (row >= 0 && row <= ds.Tables[table].Rows.Count)

                {

                    return ds.Tables[table].Rows[row][column];

                }

            }

            return null;

        }

    }

}

If you’ve never seen an extension method before, the first parameter will appear strange to you. The this keyword is added to it to show that this Get() method works on the DataSet class. Although this method is defined as a static method on a class called DataSetExtensions, it is recognized as an instance method on the DataSet class.

Intellisense even recognized my method, and the code comments I’ve added. The only difference between an instance method and an extension method from an intellisense point of view, is that you’ll still be able to notice the this parameter in the signature of your extension methods. Of course, the same goes for .NET framework defined extension methods, like the LINQ Where() method, which also displays the this parameter.

Extension methods and the Open Closed Principle

Doesn’t that look cool? I think so. When I first read about extension methods I immediately thought of a whole bunch of functionality that I could add not only to our own frameworks, but also to open source and  even the .NET framework itself. But before bloating out lots of code, let me give you some caution.

My biggest worry was whether extension methods would allow me to break existing framework code. There’s a well known principle that needs to be examined here, which is called the Open Closed Principle (OCP). In short, this principle states that classes should be open for extension, but closed for moderation. In general I think this is a principle well worth living by. So, the big question is: do extension methods allow me to change the behavior of existing classes? What happens if I write an extension method, such as the one below?

namespace Extensions

{

    public static class ObjectExtensions

    {

        /// <summary>

        /// Testing whether I can override the ToString() method.

        /// </summary>

        /// <param name="o">The object</param>

        /// <returns>My own string.</returns>

        public static string ToString(this object o)

        {

            return "Sander is always right.";

        }

    }

}

The question is, by writing this simple extension method, did I just override the ToString() method for all classes in my applications? Surely, such extension methods would kill a lot of projects really fast. Of course the .NET architects do not allow you to do so. There’s thing or two you should know about how extension methods work, and about how they are compiled.

Compiling extension methods

First, they’re defined within the context of a namespace, Extensions in the example above. To use an extension method on a class in your code, you will need to add a reference to the namespace that contains the extension methods. Without this reference, the method don’t even show up.

Second, and even more important, extension methods are in fact just static methods on a static class, as can be seen in intermediate language (IL). Extension methods only appear in your code editor as instance methods on the target class. The fact that they’re defined as static methods outside of your target class means that in an extension method you can only get to the public properties the target class exposes, but you will not be able to reach the protected and private fields of this class, nor will you be able to define new properties or fields.

Even more important, it is interesting to notice what the compiler does with your extension methods, especially in the case that you would define a method with the same signature as the method in the target class, like with the ToString() method above.

The compiler follows a straightforward rule: extension method always have a lower priority that real instance methods. As a consequence, when the compiler tries to resolve a method invocation it first looks in the target class for methods with matching signature, and consequently in its ancestors, and bind to this method if located. Only if no instance method is found, will it start to look for extension methods with the same signature. Following this convention the ToString() method I’ve defined above will never be called, as the object class defines a method with a matching signature.

Even when I try to create a (very stupid) extension method like the one below, targeting the Equals(string s) method just for matching with a string, the compiler will still bind to the Equals(object o) method from the object class, even if it has a wider scope – it compares with an object, not a string.

public static bool Equals(this object o, string match)

{

    return match.Contains("Sander");

}

This compiler directive satisfies my concerns. Using extension methods, you will not be able to alter any existing functionality. You can not override existing methods, nor get a hold of private or protected fields. In fact, the extension method totally live outside of the target class. So, you’re quite safe.

No worries then?

Well, there’s something else that concerns me about extension methods. The most demo’s I see on extension methods worry me. There seems to be no end to people extending the String class. I’ve seen demo’s that check whether a string is a valid email address, a valid URL, and even extensions that speak out a string using a synthesizer – although I expect good craftsmanship, not really something I long for in my applications.

It’s just that stuff like this should not be extensions to the string class. Unless at least 50% of all strings you use in your code are in fact email address, I would consider a different pattern here, such as defining value objects of type Email, Zip or whatsoever. In general, my concern with extension methods is that it’s such a neat little technology, that it will invite people to use it for all the wrong reasons – just because it’s fun to define an extension method for instance.

Before you know it, the developers in your project will have defined loads of extension methods, to all kinds of framework classes. You should be well aware of the possibilities this offers to break your application architecture and create lots of undesired dependencies. Just take a quick look at the following example.

using System.Workflow.Activities;

namespace Extensions

{

    public static class CodeAcitivityExtensions

    {

        public static bool Kill(this CodeActivity activity)

        {

            activity.Dispose();

        }

    }

}

Here, I’ve added an extension method to the CodeActivity class, which resides in the namespace System.Workflow.Activities. Now my Extensions namespace has a dependency on this namespace, and moreover, also on the underlying System.Workflow.ComponentModel namespace.

However, my previous code example where I extended the  DataSet class also resides in my beautiful Extensions namespace. So from this day forward, every time I use my Get() extension method for DataSet, I’m indirectly also connected to the workflow namespaces. Just as an example.

Furthermore, when you define an extension method to extend a type delivered by a framework, you also run the risk that your extension method will cause your code to break if the implementation of that type changes, or if your project migrates to a newer version of that particular framework, of .

Extending framework base classes

On the good side, you will more likely use extension methods that other developers have written, than define your own. Here, extension methods are a powerful features, as applying extension methods does not require any additional knowledge. Most extension methods will be written by frameworks developers, who now will feel enabled to solve anomalies. Or even more likely by developers who feel the need to extend base classes from existing frameworks, both from Microsoft and open source, such as in the following example, that extends the open source frameworks CSLA.

One of the most reused base classes in the CSLA framework is called BusinessBase. It serves as layer supertype for all domain objects in applications built with the framework. The BusinessBase class has the ability to validate its business rules. To this means it stores a broken rules collection that contains references to business rules that are violated during a validation of your domain object.

In some cases, for instance when displaying violated business rules on screen, it can be handy to know which business rules were broken per property. The BusinessBase class however, does not have this ability.

public static IEnumerable<BrokenRule> GetBrokenRules(this BusinessBase bb, string property)

{

    var queryableList = new List<BrokenRule>(bb.BrokenRulesCollection);

    IEnumerable<BrokenRule> result;

    if (!string.IsNullOrEmpty(property))

    {

        result = from brokenRule in queryableList

                 where brokenRule.Property.Equals(property)

                 select brokenRule;

    }

    else

    {

        result = queryableList;

    }

    return result;

}

The code example here displays an extension method for BusinessBase that loops through the total collection of broken rules using a LINQ query that filters the broken rules for the selected property (using the property’s name). This method is very useful and powerful example of how developers can tweak the functionality of framework classes. It should come as no surprise that many more such extension methods will become available from the community soon for popular frameworks such as Microsoft’s Enterprise Library, nHibernate, ADF, CSLA and of course for the .NET framework itself.

Handle with care

Extension methods are a language feature that does not really stand out in the crowd, but can be very useful, especially to avoid repetitive work that should have been a feature of frameworks you apply to your projects. However, when developing extension methods you will need to be very aware of the side effects described above. I therefore recommend to implement extension methods sparingly. Under normal circumstances, you will still extend existing types by inheriting from them, even though this requires your developers to use your new class by convention instead of an extended base class. To leave you with a proper quote: “Creativity is a natural extension of our enthusiasm.”

Let’s suppose for a moment that it’ is 1986. And let’s suppose you are starting a new company. A company that sells products or services to customers, as most companies do. Let’s say you selling (surf) board wear. You start selling your first t-shirts and you decide that you need to automate stuff. So you create a list of your customers in Excel, and type orders and bills in Word.

Someone’s father-in-law

After a while business is picking up, more orders are coming in, more boards, t-shirts and shorts are sold, and customers are even returning to your shop. This is where things are starting to get a bit nasty. You are starting to see the need for real automation. So you start asking your friends, until eventually you find someone who has some knowledge of Microsoft Access, mainly because he is using it to print address labels for his christmas cards.

A short while later your new developer, in most cases some friend’s father-in-law, a former accountant, has built you a simple application to maintain customers, orders and delivery statusses. It’s now 1990.

At a regular basis your thinking of new stuff to sell, and worse, of new ways of selling stuff, perhaps even through the internet. This poses new features to your Visual Basic for Applications written back office application. Hence the application grows and grows. Because software architecture is light in Microsoft Access, and software architecture knowledge is not apparently present in your developer’s skills, the code grows in all directions.

Three-table-and-a-screen

Now we’re back in 2010. Because business is still going strong, your 1986 back office application is still there, although it has grown from a simple three-table-and-a-screen maintenance thingy to a code base of over a million lines of code that interacts with external parties for billing, the cash registers in your shops, and also does your enterprise resource planning, including stock and shipments.

 
Three-tables-and-a-screen

And although your friend’s father-in-law still oversees all of the code, implementing the new features you need for your evolved business is getting harder and harder. “Hey, we need to be able to deliver goods in Belgium and Germany too,” you claim. The developer sighs. “Phoe, that is a major change to the system. It’ll take me about three weeks.” With every bug, change or new feature you hope to introduce, the productivity of your developer lowers. Moreover it is due to these changes and new features that always need to be introduced fast, that the software architecture of the system has never been upgraded to facilitate some layering, introduction of patterns, or even migrated to a more serious development platform. There’s just never the time to do it.

Turning 75

To cut a long story short, these days that is not even your biggest issue. The biggest problem is that the only person capable of maintaining the code is still your friend’s father-in-law, who grey haired, these days, is seriously thinking of retiring. Or even worse.

Alzheimer Architecture 

Recently I visited a customer who needs to rebuild her current back office application – and yes, it is written in Visual Basic for Applications. And she needed to rebuild the application really fast. Why? Well, the single developer of the system who had uphold the code for almost two decades is not only thinking of retirement, but actually did retire ten years ago. He had recently turned 75, and was now moving into a home for the elderly.

No cure

Unfortunately, there’s no easy way out of this rat hole:

• Automated migration will not do the trick, as there’s so many lumps and bumps in the software, that automated tooling will never identify all the finesses in the code.
• Introducing a packaged applications, such as Micrsoft CRM, Salesforce.com, or even SAP or Oracle, is not only expensive, but how do you capture all exceptions and loop holes you’ve introduced to satisfy your customers in such a package?
• Offshore development is hard too because your whole company, which likely has grown from a happy few to over a hundred employees, is depending on the system. How do your guarantee quality? And who do your trust?
• Rebuild the system yourself in .NET or Java is also not a very plausible option. There’s no expertise in the company, and moreover, you will need the domain knowledge of your employees who are always very busy with their regular work.

This is when you realize your suffering from the Alzheimer Architecture anti-pattern. It’s a progressive anti-pattern. With each business discission you take, the problems grow, and there’s just never time nor budget to solve it properly. It is as it goes with Alzheimer’s disease. Unfortunately there is no cure.

Despite misunderstanding and resistance of his space-flight loving people president Barack Obama recently aborted the Constellation space program. The Constellation program targets at putting humans on the moon again, for the first time since 1972, with the ultimate goal of possibly planning a manned trip to Mars in 2025.

Although I had never heard of the program, after reading the news on his decision I understood Obama perfectly. As a software developer his motivation to stop this over-ambitious project makes perfect sense to me. The Constellation program is extremely complex, and several deadlines and budgets have already been exceeded, apart from the fact that it is unclear whether the final goal can actually ever be achieved. As Obama said: "Constellation is over budget, behind schedule, and lacking in innovation.”

Mammoth projects

Doesn’t this sound familiar to you? Any of  us computer professionals will at some point in our career have been near or even in one or more of these mammoth projects. Over-complex, a repetition of earlier moves – similar to a project replacing an outdated system by a newer system that will deliver exactly the same functionality. Putting a man on the moon? Didn’t we do that already way back in the 1970’s?

And don’t forget setting incredibly unrealistic goals for the project.  Obama’s predecessor Bush originally offered the proposal as: “Constellation sets out to vastly reduce the costs of further space exploration." In the long run, planning a manned flight to Mars in 2025 is not much different than an international bank starting a 25 million Euro project migrating all of their mainframes before 2020 with the ultimate goal of “vastly reducing the costs of further IT exploration.” We might at some point in time reach these goals, but it’s very hard to predict when, and at what cost.

And, last but not least,  you will of course have recognized the endless expiring deadlines and budget overruns. No example needed here.

The big question

Still, the big question is: who will stop a running (and failing) project when millions have already been invested in it? Weren’t we already 80% finished? Perhaps true – but weren’t we already finished 80% one-and-a-half years ago?

With the current state of the economy, and the pace at which new and changing requirements and technologies fly by, organizations should definitively re-think. Where do these over-ambitious mammoth projects actually lead us? On average, such projects seldom deliver more than 20% of the originally planned requirements, but at multiple times the original costs. Quite simply, we can no longer afford ourselves these lingering multi-million Euro spending projects that don’t deliver.

Humility

Maybe it’s time for some humility when it comes to setting ambitions in IT projects. It’s about time we set more realistic, realisable targets, perhaps be a bit more agile, and especially simplify and standardize our IT projects and architectures. It is like in literature: writing is deleting.

I just wished more IT project managers acted like Barack Obama.

Again doing a lot of talks this spring on a wide range of subjects, from new technology, via enterprise agile to model driven development, but also about smart use cases, domain driven design, UML, and software architectures, design patterns, frameworks and .NET.

This season’s highlights? Not a difficult choice: doing talks both at Microsoft DevDays and TechEd North America is great. But revisiting SET in Zurich is not bad either. And what about the Dutch Testing Conference?

Check out my spring talks below (with links and logo’s):

• Around the world in 80 slides.
  February 9. Keynote at Capgemini TechEd, Utrecht.
• Pragmatic modeling using UML and beyond
  March 3-4. Two-day workshop at IT Works, Antwerp, Belgium.
  www.itworks.be
• We’re not in Kansas anymore. New trends in Microsoft technology
  March 8. Capgemini Micrsoft Practice Meeting. Utrecht.
• Pragmatic modeling using UML and beyond
  March 10-11. Two-day workshop at Array Seminars, Hilversum.
  www.arrayseminars.nl
• CIO challenges. From enterprise agile to model driven development.
  March 16. Talk Capgemini customer event, Utrecht.
• How smart use cases will change your life
  March 23. Guest lecture at Hogeschool Utrecht, Utrecht. .
• Agile development in everyday practice
  March 24. Seminar at Array Seminars,  Nieuwegein.
  www.arrayseminars.nl
• Do’s and don’ts in implementing extension methods
  March 30-31. Talk at Microsoft DevDays 2010. Den Haag
  www.devdays.nl 
  .
  
• Beyond agile testing
  April 7. Talk at Capgemini Testing Community Event, Utrecht.
• Beyond agile testing
  Arpil 21. Talk at Dutch Testing Conference 2010, Nieuwegein.
  www.dutchtestingconference.nl
  .
  
• Mission impossible? How to survice agile SAP SOA projects
  May 18-20. Talk at SET 2010. Zurich, Switzerland.
  http://www.sigs-datacom.de/set/set-2010
  .
  
• Estimating with smart use cases
  May 26. Seminar at Array Seminars, Nieuwegein.
  www.arrayseminars.nl
• Software architectures and patterns in .NET.
  June 2. Seminar at Array Seminars, Hilversum.
  www.arrayseminars.nl
• How frameworks can kill your project and how to avoid getting killed
  June 7-11. Talk at TechEd US 2010, New Orleans, USA.
  northamerica.msteched.com
  .
  
• Pragmatic modeling using UML and beyond
  June 23-24. Two-day workshop at IT Works, Antwerp, Belgium
  www.itworks.be

Smart use cases are a great technique for specifying standardized requirements in many types of projects. Over the past few years we have smart use cases being modeled and written in projects using Java and .NET, as you might expect, but also in Sharepoint projects, business intelligence, service oriented projects and even SAP implementations.

Stereotypes drive standardization

Standardization of your requirements can be an important driver to simplify software development and software rejuvination projects. An important addition to the smart use case modeling technique is the use of smart use case stereotypes.

These stereotypes represent often occurring patterns of requirements. Well known our the stereotypes manage, that represents a maintenance use case on some domain object, or search, which helps users to find a particular domain object, guided by search criteria. Other known stereotypes include select, master-detail, view, and report. As you might expect, these stereotypes all deal with user interaction.

Over the years however, we have gathered quite a few stereotypes, also in other fields. We have described for instance stereotypes that deal with Extraction Transformation Load (ETL) processes in data warehousing projects, such as aggregate, calculate and load.

Stereotypes not only help to put focus in modeling workshops, they also facilitate more easy project estimates, and even allow for straightforward code generation and isolated testing. Even better, once you marked the stereotypes (and put them into a UML profile), it will allow you to create much richer use case diagrams, modeling in colour.

Modeling in colour using (profiled) smart use case stereotypes

Slicing service oriented projects

More recently we have applied smart use cases to service oriented projects. These projects tend to be more complex than regular software development projects, and focus around more elaborate architectures Service oriented projects mostly include:

• Flexible front-end features. These can be either web pages, mobile phone front ends, Windows native applications or even Flash or Silverlight applications.
• Process-oriented middleware. Service oriented processes always implement business processes. These business processes or workflows are run in middleware, more specifically in an enterprise service bus (ESB), although having such a bus is not mandatory. There are many vendors supplying busses, such as IBM, SAP, Oracle and Microsoft.
• Stable services. A very important part of service oriented projects consists of the discovery and realization of services. Most often these services are implemented on top of existing legacy or packaged systems.

These slices in service oriented project all serve particular means and responsibilities. The approach to modeling smart use cases in such projects offers quite a few benefits. Next to modeling the front-end use cases, smart use cases help to model the business processes, and discover the required services.

Stereotypes for service oriented projects

In recent projects, that involved such complex service oriented architectures, we have gained a lot of experience in identifying and modeling smart use cases. This week we have analized a number of such models from different projects. This results in a set of new smart use case stereotypes, specifically targeted at the processes that run in middleware, and the encapsulation and execution of services.

One of the smart use case diagrams we’ve investigated

Next to our well-known front end smart use case stereotypes, the new service oriented stereotypes include the following: process, aggregate, dispatch, validate, read, write and inform. I will elaborate on these stereotypes in the next sections.

Stereotype process

A use case of stereotype process executes a business process. In most cases it delegates the execution of services to other use cases, often of types aggregate and dispatch.

The process use case is in the lead, and uses the post-conditions from called use cases to steer the process. This type of smart use cases is generally implemented in the enterprise service bus, such as BizTalk or SAP Xi.

Stereotype aggregate

An aggregate typed smart use cases generally is used to create a single response to a request.

The stereotype is called aggregate as these use cases trigger other smart use cases, othen of types dispatch and read, and aggregates the results from these use cases into a singel response.

Stereotype dispatch

Smart use cases of type dispatch are intended to link the aggregate or process types use cases to services that either require some mapping, or typically run outside the organization.

Mapping is often required, as the available services do not always deliver what is required by the calling use cases. The dipatch smart use cases performs the mapping (hence and forth) and relays the work to the actual service, that is often of type read or write.

Stereotype validate

Executing business processes often include specific validation services, for instance to see if an customer is allowed to add a particular product to their subscription (as in one of the projects we investigated). Smart use cases labeled with the validate stereotypes return a single answer, that can be either a yes or no response, or a single value from a limited list of possible codes.

In cases of more complex validation, for instance when match-making between different back-end or external systems is required, the validate use case relays fetching the required information to dispatch,  read or even aggregate smart use cases.

Stereotype read

Smart use cases with the read stereotype are the most basic services. Here one or more domain objects or entities are returned from a back-end system. This often includes compressing a whole hierarchy (such as with many SAP systems), such as a product hierarchy into a single returning message.

Stereotype write

In reverse, write typed smart use cases will pick up such a compressed hierarchy that is input to the use case, distribute the hierarchy into the back-end systems and persist it.

Stereotype inform

During the execution of a business process, the actual user of the whole process (being the actor of the user goal level use case initiating the process) needs to be informed of the status. The actor might get an email, such as in a subscription process to acknowledge successful subscribing, or be sent a letter. Other alternatives are of course also possible.

The inform use case is responsible for merging the text of the message being sent with the actual domain objects the use case model evolves around.

The Accelerated Delivery Platform’s (ADP) Agile Dashboard is a pragmatic and publicly available tool (free) for managing project progress online. The Agile Dashboard was originally intended to manage progress for our agile projects, but these day it is used in a much broader perspective.

As the ADP Core Team receives a lot of questions about it, it’s time to present a small introduction here, including future directions.

Administrating projects

The Agile Dashboard simply administrates projects, project teams, work packages and work items. Furthermore, the Agile Dashboard allows you to define statusses your work items go through. In a typical Scrum project you will likely have ToDo, Working and Done. In a typical Smart project this will likely be New, InIteration, Working, Testing, Rework, Done, Dropped.

Managing work items

Work items such as user storie or features can either be added manually, or via a straightforward import feature that allows UML models to be imported in the Dashboard (from XMI). As Smart projects evolve around implemeting smart use cases, use cases can be marked as work items automatically. For each work item the complexity of the item needs to be established. Whem importing a UML model using smart use case stereotypes, establishing complexity is automatically handled.

The actual Dashboard

The main page in the Dashboard shows the progress of your project’s work items, or a the work items in a single work package, as shown below. This page will give you an immediate overview of the project.

When a work item changes status, e.g. when development is done, the user can drag-and-drop the work item to te next status, in this case for instance Testing. This pop-ups a next page where the user can add comments, and give an estimate for the amount of work still needed on the work item. Note that this is typically an action that is done during stand-up meetings.

Real-time burn-down

This information is used by the Agile Dashboard to recalculate the burn-down chart for the project real-time.

The freely available Agile Dashboard offers very basic functionality and does not excel in abundant charting and progress reporting. This is done to make managing progress in projects as easy as possible – albeit the Dashboard could do with some minor useability improvements.

Using the Agile Dashboard thus presents real-time project progress. It’s online availibility makes it suitable for distributed scenarios, e.g. in offshore – we e.g. manage projects with it that are executed on several locations simultaneuously in the Netherlands and India.

Adding one and one

The current version of the agile dashboard, although of great use to projects, is somewhat limited, as it was not designed for (branded) use by different organization. True multi-site, branding and extensive reporting therefore lack, as does a rich internet user interface.

We are currently in the process of re-modeling our Agile Dashboard (new more sexy name pending). We think it should implement the following features:

• Autorization using open mechanism, such as OpenID.
• First use by new organizations granted by administrator to avoid fakes, possibly using Yammer-like mechanism.
• Organizations can set up their own projects and project teams, similar to current version of dashboard. Possibly allow branding.
• Organiszations can set up template projects, we currently supply both Smart and Scrum, but can easily also support RUP or even linear development.
• Organizations can set up their own estimation scale, we currently supply the smart estimation scale.
• Organizations can set up their own product life cycle, we supply smart use case life cycle, similar to current version of dashboard..
• Import of work items / smart use cases from Excel, UML XMI. Organizations can set their own work items types, similar to current version of dashboard..
• Project dashboard with drag-and-drop work items progress, similar to current version of dashboard.
• New, more user-friendly user interface, built in Silverlight.
• Real-time burn-down charts and other reporting, similar to current version of dashboard, but realized using Silverlight charting library.
• Simple mobile client for tracking work item progress, possibly even an open service API.
• Possibly integration of dashboard with Team Foundation Server.

We hope to start working on the new version real soon.