The slides and recording of our (Jezz and myself) session ‘Build Your Own Software Factory’ are now available for download.

From the statistics of my blog I know that most of my ‘readers’ don’t live in Japan, but If you ARE in Japan on 21st-24th August and plan to visit TechEd Yokohama, come join us and attend our session.

Factories 201 Series - Building Software Factories

• What are they (concretely)? (Edward)
• When would you build one? (Jezz)
• What do I tell my Manager? (Edward)
• How long will it take? (Jezz)
• What would you build? (Jezz)
• With what would you build it? (Edward)
• How would you build it? (Jezz)
• What should you strive for? (Edward)
• What are the challenges? (Jezz)

Post authors: Edward Bakker and Jezz Santos.

Well, we have finally come to the end of this 201 series on ‘Building Software Factories’.

First time arriving here? - we have listed the entire series above – please have a look through those posts before ending it below.

If you are arriving here after reading (hopefully all) the posts in the series, we wanted to say, good on you, and thanks for sticking it out with us.

Series Wrap-Up

Well, we have spent quite a bit of time and thought in putting this series together for you. The motivation behind this effort has been that we’ve recognised that there is little practical information helping ordinary professional developers on getting started with building and understanding software factories. We have had quite a head start on this and wanted to share our knowledge and experiences with you and the community to promote the uptake of building factories, which in turn should promote the adoption of software factories and the industrialisation of software in general.

This series was created in a format that asks a logical sequence of questions that you might have when trying to figure out how to build software factories today. We have covered many such issues as they arose and shared much of the combined knowledge and experiences built up over the last few years in this space. We will be sharing more resources on this in future posts – so stay tuned to our blogs.

We hope that you can take the guidance represented in this series forward as a starting point for developing your own software factories further. We recognise openly and frankly in the series that this is not a trivial undertaking today, but hopefully we have shown you that it is very possible and how to successfully approach it. Hopefully we have also directed you to what’s most important, where you should direct your energies, and how to build factories practically and realistically.

Your Feedback

One of the primary objectives of this series was to connect with many more of you in the community and share ideas and innovations about everyone’s efforts at building factories today.

We have received a small number of comments to this series so far, which we have been keen to discuss at length, and hope we have done a good job at answering the questions appropriately. Hopefully, there are many more of you out there building factories, or at least considering building them, and so we hope to hear more from all of you on that. Please drop us a comment on the appropriate post and we will work hard to get you an answer you will find useful.

If we are doing a good job please let us know. If we are doing a poor job and you don’t like the guidance we are giving please let us know that also. This should shape any future guidance we give the community in this space, and at least we can start further discussions with those of you in this space. Hopefully, we will also get a feel for just how many of you are out there are in this space.

The Final Say

There should now be a little more information out there to help you build a software factory right now on the Microsoft platform with the current toolsets. This series was created exactly for that purpose to share with you the experiences we have had in building factories and give you insights into the issues we have faced on this road already.

As more and more people jump into this space we expect to see from the community more detailed, formal guidance on this subject, so that the community as a whole can learn from previous experiences and elevate the adoption and the application of software factories to real world projects.

After all we think that the act of building a software factory should leverage the tenants that software factories promote themselves, i.e.

“Package domain expertise and guidance in a form that others can re-use and manipulate to create solutions using simplified descriptions of the problem at hand.”

Hopefully we have done a good job of that for you, and hopefully you will consider the experiences we have shared in this series in your factory building efforts.

Come see us...

We are proud to announce that we will be presenting a session on ‘Real World: Building Software Factories’ at Tech-Ed 2007 in June this year.

The 300 level session will contain material from this series and demos and examples of other factories to explain the concepts. We would love to meet up with you there and answer your factory building questions.

Many Thanks, Jezz Santos and Edward Bakker

Previous posts in the Factory 201 series:

• What are they (concretely)? (Edward)
• When would you build one? (Jezz)
• What do I tell my Manager? (Edward)
• How long will it take? (Jezz)
• What would you build? (Jezz)
• With what would you build it? (Edward)
• How would you build it? (Jezz)

After reading all of our previous posts (don’t tell me you skipped one!) we know what automated assets to build, which tools and technologies to use and how to build a factory. The final step here will focus on the quality attributes we have to keep in mind during the design and development of our factory in order to make it a stunning success! In other words, “What should we strive for?” when building our factories?

Abstraction

In a previous post “With what would you build it?” we already mentioned the power of using abstractions in our factory. Let’s dig in this a little further and see why abstraction is definitely the top thing we should strive for!

Increased Productivity

One of the advantages of using abstractions in our software factory is significantly increased productivity. We can think of an abstraction as a simplified representation of a specific problem. In fact, we are probably using abstractions in our job as a developer or architect every day. (i.e. 4G programming languages, class diagrams, API’s, class libraries, frameworks, DSL’s etc.)

Think about it, for example, if we build a helper class to perform a specific task (say: logging to a file) we are actually using abstraction. We create a class and some methods that simplify the details of logging, file management, rolling-over, naming, log details and formats etc. Once we have the helper class in place we can just re-use the helper class using some basic parameters without focussing on the details of how it works anymore. We solved our problem more simply by creating the helper class.

Now, let’s elaborate on this a little further. Suppose we only use this helper class in certain circumstances of a particular type of solution, let’s say only for certain components of a particular application. In this case, we don’t use or need all the possible parameters of the helper class (because we only use it in a certain way in this context), and let’s say we identify 5 different cases of use, each with different parameter values. Now we can build another higher-level helper class that wraps the original helper class, and defines only one parameter which describes one of our 5 cases – perhaps an enumeration. Our new helper class is now an abstraction of the older helper class – because it simplified its usage in this particular context. Typically, this new level of helper class becomes part of a framework for those types of applications, because we have made certain types of assumptions about how the helper class is to be used.

[To clarify this, using abstraction for simplifying a problem is something we see all time. For example, take ADO.NET that hides a lot of the low level details that are needed to communicate with a SQL Server database. On top of that we have Enterprise Library that adds another abstraction level to make data access even more straight forward. In the future, ADO.NET vNext and the Entity Framework will do an another attempt to raise the level of abstraction for “data programming”]

As we continue to build up the levels of abstraction, you can see that we can solve more specific problems at less level of detail, and also by using a higher level description or language. If we can get close to a point where we can provide these abstractions in terms of the specific problem we are solving, we are making it real easy to program our application (based on minimal information). This is where productivity is gained.

Abstraction Roadmap

When developing a solution you can think of many abstractions, when you add modelling and automation to this picture the possibilities for abstraction build upon each other vastly:

Platform API’s → Application API’s →Helper Functions →Class Libraries → Frameworks → Domain Models → Software Factories

Each of the abstractions denoted above increases the level of abstraction of the problem you are solving, and in each case also narrows the domain of the solution you are providing for – this is the simplification part. I am sure it’s obvious to you that using a domain model to configure an application is far quicker than writing programming API’s to describe the same thing.

The other key point here is that I can configure my application in terms of high level things that make sense to that application and not in the low level details of the API’s. This means that those using the domain model now don’t need to be programmers they can have higher level skill-sets like solution design, requirements gathering and interpretation etc. By raising the level of abstractions we are moving from a physical design (platform API’s) to a logical design (domain model) that exposes less details and is understandable for a much wider audience.

Automated Guidance

Another very important aspect we have to keep in mind during the development of our factory is the level of automated guidance we expose to our factory users. In our post “What would you build?” we have already discussed the different types of automation assets that we might want to use in our factory but we have make sure we use them in their most effective way.

Remove Mundane Tasks

A top quality attribute (from a developer’s perspective) for using our factory would be to provide automated assets that remove the rote and menial tasks from the factory user’s ToDo list. At best remove these types of duties completely from the developers ToDO list. We as solution developers, often have some predictable boring tasks that we have to do over and over again. By exposing automated assets for these types of tasks we save our factory users from completing these mundane and in-variable tasks and let them focus on the variable and more exciting design aspects of the solution they are building. Our factory users will definitely love us for that!

Transfer Domain Knowledge

The design of our automated assets must ensure they capture knowledge and experiences and can be used to make that knowledge available for others. Using the example of a DSLs’ domain model which holds the knowledge about the underlying solution domain and how it can be and should be manipulated. By using the DSL, the user is presented with all the knowledge of how to describe the domain, the parts and relationships of that domain, and the description of the pieces within it.

The important part of this is that the domain model helps the user to figure out the key parts (concepts) in the domain. The domain model (and designer in case of the DSL) guides the user through the process of modelling a solution for the problem by using concepts that make sense to him. In fact, the user learns the domain and the important concepts of it but also how to provide a solution by using these abstractions.

Enforce Compliance and Completeness

When we design the automated assets (and factory as a whole) we encapsulate the architecture of the factory product in the domain model that is used by the factory – this is the model of the product.

We can build automation assets to populate or configure the product model and in this way ensure compliance with the implicit architecture of the product. Because these models have configuration to address their variance, validation can be used explicitly to ensure the factory user has not configured parts of it incorrectly with respect to the whole solution.

Automation assets can also be used to examine the model, check for completeness and use that information to guide the factory user in his next steps he needs to execute in the factory to complete the product.

Full Automation vs. Flexibility

When designing our automated assets we have to keep in mind to keep them limited and focused on one particular concern. It doesn’t make sense to build a huge fully automated asset that covers multiple concerns and generates a lot of unrelated artefacts (i.e. source code and configuration files). It also doesn’t make sense to build a DSL that tries to model to world. If we expose a number of small, highly focussed automation assets we have a much more flexible factory because we can, for example, change the order in which we manipulate the assets, combine them or even decide not to use some automation assets at all to better meet the requirements for the product we have to build.

However, if we go too far with variability and refining our abstractions and automated assets to a more detailed level we might end up with a factory that is less productive because there are more “moving parts” that needs to be configured which also increases the complexity of using the factory.

Finding a balance between full automation and flexibility is absolutely key here!

Usability

Because we are so busy developing our software factory we might forget that building a fully fledged and super cool software factory that generates the perfect product isn’t enough to make everyone want to use it. In the end it is often the factory user that has a huge influence in the adoption of our software factory. This means we have to make it cool to use the thing and make it flexible and intuitive enough to use and improve the experience of building solutions in this domain with it!

You don’t want people to discard you factory because it too hard to use, or discover how to use effectively. Don’t assume they will read the manual!

Promote Adoption

Another thing we have to keep in mind during the development of our factory is the fact that the factory is likely to be used by other less-skilled and less-knowledgeable developers than the ones that building the factory. They might not even have to be developers! This means, we might have an issue here in convincing other development teams to adopt the factory and use that instead of their existing tools, frameworks and utilities they are so familiar with. Or worse, have built themselves!

To get our factory adopted by our audience we have to lower the bar for them to start using the factory. Of course there are multiple ways to achieve this and they will probably differ dependent on the organization structure we are working in. Suppose we are working in an organization where the factory is build by a dedicated team that is known by the several project teams responsible for delivering products. In this example, the factory team can support the project teams in the start up phase of their project to get familiar with the factory and makes sure the project team gets up to speed on using the factory real quick. Once the factory is adopted by the project team responsible for building the product, the factory authoring team can start collecting the feedback from this team to enhance the factory and increase the usability of the factory even better. You see this in the process described in “How would you build it?”

Role Sensitivity

A software factory, just like any other development tool, is potentially used by a lot of project team members (factory users). i.e. Developers, Architects, Testers, User Experience Engineers, Technical Writers etc. To get the full potential from our factory, and factory users, we must make sure the factory targets the audience that is supposed to use the factory in the way they want to use it. In other words before we start developing our factory we need to know what types of users we expect for our factory. Do we only target architects, developers, testers or is it a mix of these types of users that we are aiming at?

Why is it important to know the factory users? Well, each type of user is interested in a different set of concerns and views of those concerns (In factory terminology this is called a ViewPoint). For example, an architect might only be interested in the high level view of the system design while the developer is interested in much more technical details.

Once we know our audience, and those who we are building the factory for, we can attempt to provide them with the right tools, views and level of abstraction to support them do their job in the best possible way and be more productive when using the factory.

Customization

Software development is rapidly changing. New technologies, new paradigms happen almost every day. To be successful in software development we have to keep up with these changes, and therefore our software factory has to too!

In our post “How long will it take?” we have seen that our factory only start to pay back after we have used it for greater than ~5 times, so we better make sure we can customize our factory very easily to meet evolving requirements and additional product variants. This is exactly why we have an iterative development cycle as defined by “How would you build it?”

One of the key requirements of any development tool like a factory is to be flexible enough to address as many use scenarios and development environments as possible. Otherwise its’ use is limited. There are two aspects of customization worth noting here:

Factory Customization

Firstly, there is customization of the factory itself, usually to accommodate a change in architecture or change in variability of the existing architecture that the factory models. This is hard to accommodate if the factory itself presents a closed programming interface. However, this can be accommodated by defining good extensible infrastructures to the architecture so that those using the factory can easily customize it to add or remove features or variability that the factory presents. We see this most when ‘horizontal factories’ are customized to suit certain vertical scenarios, where variability in most cases wants to be predefined, and fixed, and therefore requires removal from the user experience of the factory.

As an example. Consider a (horizontal) factory that builds web services. If I want to ‘verticalise’ this factory and customise it to suit building certain types of financial web services, then it’s likely that for financial web services, I want to pre-define certain Operations Contracts and Data Contracts to use. The Operation and Data Contracts are one point of variability of the horizontal factory, and the horizontal factory provides assets to allow the user to configure them. Now that I want to customise this factory to make a new verticalised version that is financial services based, I want to be able to say that the Operation and Data Contracts are fixed to certain industry standards, and no longer configurable by the factory user.

Product Customization

Secondly, there is customization of the automation assets used to create the product itself. Typically, the scenario here is that you want to alter the automation assets used to transform the model of the product into a physical solution.

As an example. Consider a factory that generates source code as its final product. The source code is determined by the factory in code templates, which are used to transfer the models of the factory into a physical solution. So consider, that the generated code does not agree with you in some way. Often, this needs to be changed to suit an organisations’ coding standards, or include some organisations standard copyright header, or perhaps you want to change the code that is output because you have a more optimal way to write the solution.

Extensibility & Composability

One of the key issues of building factories is in providing the right guidance and solutions to varying requirements. Creating a factory for a large domain increases the chance that the guidance it offers is limited in addressing wider usage scenarios.

For example, consider a factory that builds web services. This factory is likely at some point to define a mapping of the web service’s logical architecture to source code artefacts. In doing that, the factory itself will have to make decisions about what patterns and technologies to use to do that. The assumptions it makes to do that may not be valid for all cases, and it is such cases that may prohibit the use of the factory at all, if the factory user cannot customize or influence how that is done. In smaller domains the chances of meeting this issue are smaller, because the possible solutions are smaller in number. However, even in smaller domains, there can be several solutions that apply to different circumstances. This means that larger domains need to be composed of smaller domains, where each smaller domain could be swapped out for a different implementation.

Composite Factories

In order to practically achieve this we need to be able to compose larger domains from smaller domains, and have the domains interact with each other. Furthermore we need to be able to potentially swap-out each domain with a different implementation.

In factories, this can be achieved by providing extensibility mechanisms that accommodate plugging in and out various other factories to provide the sub-domains. We call these sub-domain factories ‘factorettes’. These extensibility points are key for expanding the use of your factory, so others could provide alternative implementations for the pieces of you factory to suit their needs.

One such use of this is to provide specific technology implementations or strategies for implementation such as the ‘EFx Factory’ does, so that factory users can pick and chose which technology to use to implement the Service Contract or Data Mapping layers, for example. Of course you can choose many other aspects to make extensible in your factory apart from technology. Perhaps you want to allow other factories to provide better views of a part of a domain, or provide better implementations of it as well.

Summary

Previous posts in the Factory 201 series:

• What are they (concretely)? (Edward)
• When would you build one? (Jezz)
• What do I tell my Manager? (Edward)
• How long will it take? (Jezz)
• What would you build? (Jezz)

To continue the thread of actually creating your factory, in this post we will answer the question "With what would you build it?" We will discuss the software factory enabling technologies available today (on the Microsoft platform) and don’t speculate about possible future tooling. In our previous post “What would you build?” we discussed what we would build for our factory, and now we will see how that is mapped and addresses by specific technologies.

Let's have a look at some tooling and technologies and see how we can use that to build our software factories, starting today.   

Automated Actions

Today's generations of software factories, like the ones available from Microsoft p&p, are primarily composed of a related set of automated activities (called recipes) that encapsulate a huge amount of best practices and domain expertise. The recipes often represent repetitive development and design tasks which result in generated source code or configuration files to help accelerate the development of our factory product. Let’s find out what tooling we can use to build these automated activities and how we can expose these, so called recipes, to our factory users.

Guidance Automation Toolkit/Extensions

The Guidance Automation Toolkit (GAT) is a toolkit that simplifies the development of automated actions into re-usable assets and provides mechanisms to make these assets available to architects and developers using Visual Studio. To accomplish this, GAT leverages the Guidance Automation Extensions (GAX) which is actually a runtime environment that expands the capabilities of Visual Studio and makes it easier to integrate and run automate tasks from the Visual Studio environment. So, we are actually using GAT and GAX together to build and run recipes and deploy them as guidance packages. ehhh...? Now what are recipes and guidance packages?

Recipes

Think of a recipe as set of automated instructions, called actions that together represent a specific automated activity or task. These actions are relatively small units of work and can be anything from creating a file, getting the value of a registry key or return the name of the currently selected project within a Visual Studio solution.

A recipe is defined in an XML file in a predefined structure that is interpreted by GAX at runtime. GAX supports invoking wizards for a recipe to gather user input that can be used as variables during the execution of the recipe to create variability in the artefacts the recipe outputs or configures. For example, a wizard can prompt the user to select a file location, ask for a name of a file that gets generated or ask the user to select items from a list. Recipes that don’t require user input can gather the variables from the environment to use in the same way. Recipes also can provide a convenient means of giving the user feedback about the actions being performed.

We can build recipes using GAT, it provides us with a special solution template in Visual Studio to author recipes, it comes with predefined solution structure, example actions and recipes and other utilities that give us a head start in building our own recipes and wizards.

Guidance Package

A Guidance Package is an installable Visual Studio package that contains a collection of recipes (and some their support classes like, converters, value providers, etc.) and is basically our deployment mechanism for the recipes that we built and want to expose in our software factory. In fact, the current generation of software factories as we see today are for the most part simply implemented as pure guidance packages.

That leaves us with another question: "how do we build a guidance package"? Again, this is where GAT comes in. Actually, GAT is a guidance package itself that helps architects and developers build guidance packages. Once we are done developing and testing our guidance package we can create a Windows installer package (MSI) and distribute our guidance package to our users.

Unfortunately, building and testing guidance packages (recipes) isn't as straightforward as we want it to be. For example, building recipes basically comes down to hacking in XML files and isn't supported by any graphical designers. Also the documentation is still limited. This makes the learning curve for developing guidance packages a little steep.

Artefact Templates & Generation

As said, some of the automated activities in our software factory will possibly result in one or more generated artefacts for our solution. These artefacts can be anything in the range from source code, XML files or a piece of documentation. To support this kind of artefact generation in our software factory we can use text templates (also referred to as "T4" templates) and the Text Templating Engine. This technology is included in GAT but also in the DSL Tools which we will discuss a little later in this post.

A text template is basically a text file that contains the (boilerplate) text you want to generate (i.e. C#/VB.NET code, XML, text etc.) that is mixed with control markers (<#, #>) and control blocks. The text between the markers is evaluated and provides the dynamic behaviour to the text template. Control blocks are used to control the processing flow of our templates. In other words they determine what text is output and how the Text Templating Engine navigates over a provided object model. This works very similar to how classic ASP work where you could combine server side directives with the raw HTML code.

Below an example of a T4 template that comes from Service Factory. The text between the markers (in bold) is evaluated and mixed with the static text in the template.

namespace <#= ServiceImplementationNamespace #>

{

   [System.Web.Services.WebService(Namespace = "<#= ServiceContractNamespace #>", Name = "<#= ServiceImplementation #>")]

   [System.Web.Services.WebServiceBindingAttribute(ConformsTo = System.Web.Services.WsiProfiles.BasicProfile1_1, EmitConformanceClaims = true)]

   public class <#= ServiceImplementation #> : <#= ServiceContractInterfaceNamespace #>.<#= ServiceContractInterface #>

   {

   }

}

The most powerful feature of text templates is that they can be invoked from recipes and they can receive variables from our recipes. In this way we can build recipes that collect input from our users or factory environment and use that as input in our text templates to generate artefacts.

A major advantage of using text templates as a means to generate artefacts is that they can be customized very easily, typically to reformat the textual output or enhance it in some way (perhaps to suit an organisations coding standards for example). Text templates are deployed as plain text files so it is straight forward to customize them.

The fact that text templates are so easy to customise makes them perfectly suited for an iterative development approach. We can start with a “hard coded” copy of the desired output and we can identify the variability points in the code. We can then pass some arguments to the template and from that point we can add control blocks to parameterize it and hook into the underlying argument’s object models. One disadvantage of a text template is the lack of designer support like syntax colouring (we added the colouring in the example manually) and intellisense. This makes writing and maintaining large text templates a bit ’un-guided’.

Solution & Project Templates

As we saw in “What would you build?” we can use the solution structure to provide guidance on how to physically relate the artefacts that represent the product of our factory. Unfortunately, defining this (physical) structure is often challenging because of the struggle to define both the deployment model and the logical architectural model of the product in this single structure. Today, the solution structure is all we have (Solution Explorer), so let’s have a look how we can implement predefined solution structures in our factories to describe the physical structure (deployment) of the product.

[The logical architecture is best left to another separate view, which as we will see later we can provide with deep Visual Studio integration]

Solution structures are captured in so called Visual Studio Templates (*.vstemplate files) that are written in XML and are expanded by the Visual Studio template engine. Visual Studio 2005 now supports the creation and packaging of these templates. However, these templates are by default static, and require a separate mechanism to gather variables from the user or environment to enhance them with dynamic values. As it happens, GAT has just that mechanism, and can bind recipes and wizards to these templates.

Let's have a look how we can use this mechanism with GAT to ‘unfold’ certain types of templates:

• Solution templates. These templates unfold an initial Visual Studio solution file (*.sln) and structure that could represent the entire physical structure of the product of our factory.
• Project templates. These templates unfold a project (*.csproj, *.vbproj etc.) within an existing solution.
• Item templates. These templates unfold new project items such as a class files, XML files or any other valid project item.

One of the primary features of GAT is automating the creation of the solution structure. GAT accomplishes this by hooking into the VS template mechanism. This means recipes are allowed to participate in the template unfolding so they can, for example, collect input from the user to customize the names of the template artefacts and take additional actions to further configure the created solution items. For example, write some settings to a app.config file. A perfect example of this is demonstrated in Service Factory where the user is prompted to provide the name of the service, which is then used in the naming of the solution and projects within the solution. We have now discussed GAT and recipes, and learned they are perfectly suited for automating tasks and providing initial solution structures. Let’s look at providing abstractions to our factory and enhancing the design experience and productivity of automating solution assembly. Recipes with wizards often do form a visual abstraction of the underlying part of the product they are automating, but this abstraction is lost when the wizard completes. Let’s have a look at Domain Specific Languages and the DSL Tools and see how they can help enhance that experience and increase the level of abstraction represented in our factory.

Domain Specific Languages

A Domain Specific Language is a custom language that targets a specific domain and helps us describe the problem by using concepts that make sense for the target domain. But how would we build one? This is where the DSL Tools come to rescue. We can use The DSL Tools, (part of VS SDK), to define our domain model and easily represent that domain model with a graphical designer. Together, the domain model and the graphical designer form our DSL that we can integrate into our software factory.  

[You can also provide a textual representation of your language instead of visually, but this is not supported currently out-of-the-box with the DSL Tools yet, unless you are prepared to define that language in XML.]

The DSL Tools also supports artefact generation using T4 templates which means we can write templates that generate (source code) artefacts directly from models we created in our DSL. The DSL Tools is just everything we need to build Domain Specific Languages today.

How would we use the DSL Tools?

So, how would we use the DSL Tools to build a DSL for our factory? Would we build one big DSL and designer that covers the complete product? The answer to this question is probably NO. Although our factory only targets one product (with several variants maybe!) it is likely that the scope of the domain is too broad to fit in just one designer (view). Therefore, we probably want to build a DSL that only targets one concern, component or aspect in our software factory.  For example, we can use a DSL to model a piece of the architecture of the product that is build in our factory or use a DSL to model one of the work products (logical components) in our factory. In this case both DSL's help us presenting the right level of abstraction to our factory user for each of the sub-problems we identified in our factory.

But wait…, we have to make it very clear why we state to keep the scope of the DSL (domain model) limited to only one concern or piece of architecture. In fact, we can think of examples, like the factory schema, that are represented in a ‘larger’ domain model. Mostly, this is because off a limitation in the current technology. The current release of the DSL Tools only supports one view per domain model. In other words, the view that is exposed in our DSL is tightly coupled to the domain model. In future releases of the DSL Tools this limitation will disappear which means we can define multiple views with each view exposing only a part of a, possibly larger, domain model (like the factory schema). Please note that this doesn’t mean we have to build huge domain models in the future, but it will give us more flexibility in our domain model (DSL) design.

[You can find more information on how to use DSL’s in the context of software factories in the article ‘Bare Naked Languages or What Not To Model’ from Jack Greenfield]

Another important limitation of the current DSL tools is that there is no easy way to reference elements in one domain model with those in another, so stitching together domain models to fit larger domains is also challenging. However, there are DSL Power Toys soon to be released called ‘DSL Integration Service’ that enable that cross-referencing.

Another thing to keep in mind is that we don't have to start with a DSL in our factory from the very first beginning. It is absolutely possible that an asset that is implemented as a GAT recipe in a first iteration of our factory evolves in a DSL in one of the next iterations of our factory development cycle.

DSL Advantages

An important advantage of a DSL in our factory is providing a simplified view (abstraction) of the problem domain and to help us to provide architectural guidance in assembling that part. By using specific languages we can present our factory users with the right level of abstraction and use the designer as a way to provide context for our actions and the artefacts we are delivering from our factory. Recipes can also be applied to these designers to manipulate the underlying domain model, and automate the configuration and creating of the elements in these models.

The domain model that is introduced by the DSL (and stored on disk) can be validated and evaluated to determine the progress, validity and completeness of the product we are building. The domain model also provides us with a means to introduce “state” in our factory. We can generate artefacts over and over again for our populated domain model, something we can't do easily with just code generating recipes, where we have to populate the wizards with the necessary information on every run.

A very good example of the usage of a DSL, together with recipes is in the EFx Software Factory, which can be seen here.

DSL Challenges

So, if Domain Specific Languages can add value to our software factories, why haven't we seen them in many factories yet? We think the answer to this question is easy. The reason for that is the lack of "out of the box" integration between GAT and the DSL Tools. Currently, it is hard to use model information in our GAT recipes, without writing the custom code to support this ourselves. Luckily this scenario will be supported soon by means of a DSL Power Toy that becomes available.  

[A more complete description of Domain Specific Languages, the Microsoft DSL Tools, together with an example of an 'real life' DSL can be found in this article]

Integrated Design Experiences

The technologies that we discussed so far are first class citizens in Microsoft's software factory offering. However, there are more technologies, which are not dedicated “software factory tools”, that can add enormous value to our software factories.

Be warned, these technologies are a little more difficult to use than those described earlier.

Tool-Windows and Custom Editors

One of the things we will notice when we are building software factories is that there will be situations where the decoupled GAT/GAX and DSL Tools features simply don't provide the experience we need. For example, we might want to provide a more integrated user experience for our factory users and decide that we need to provide the user with a larger view of the factory product.

To support this product view we might want to build custom tool-windows or specific editors. An example of this can be seen in the EFx Factory that hosts a custom tool-window that provides a ’Product Explorer’ view for the factory in addition to the default ‘Solution Explorer’ view.

We may want to enhance views with the use of customer editors that provide different abstractions and increase the productivity of modifying the view. An example of that can be seen here where we built a custom tool-window to maintain hierarchical data within a DSL.

It's good to know that we can use the VS SDK for all that. This SDK provides us with utilities and documentation that explains, for example, how to build a tool-window and hook that in Visual Studio. The bad part of this is that creating Visual Studio extensions like this isn't always as straight forward as we like it to be. Besides, building just the tool-window (or editor) isn't enough; we also have to make sure it integrates with GAX or the DSL Tools. Unfortunately this isn't something that is described in great detail anywhere so we are on our own in implementing this. 

If you are interested in this type of integration you better make sure you have the VS SDK documentation ready and find your way to the GAT, DSL Tools and Visual Studio Extensibility forums!

Distributed Application and System Designers 

One means of providing a high level view of your product is to leverage the ‘Distributed Application and Systems Designers’ that come with the ‘Visual Studio Team Edition for Software Architects’. These designers, although currently geared to deployment, provide a useful view of applications which can be composed into larger black box systems. If your factory targets the domain of applications represented on this design surface, this should be a natural choice for leveraging in describing your product, rather than creating DSL’s which duplicate this functionality.

Unfortunately, integrating with these designers today is very challenging, although it is possible to provide custom shapes using the SDM SDK (part of the VS SDK). Integrating recipes and solution templates into these designers is more of a commitment and currently not officially supported since this requires integration with a non-public API. However it is possible to a large degree, as demonstrated by the EFx Factory here. Where you can ‘drill into’ the applications on these diagrams and expose DSL’s containing the inner workings of the applications. You can also ‘Implement’ these shapes by using custom GAT project templates.

In the future, this type of integration will become a lot easier and integration between these designers and GAT and other DSL’s will be a better experience for factory builders. This will provide many factories the capability for top-down solution design.

Summary

In this post we have discussed the relevant technologies that we can use to build automation assets for real life software factories today. We haven't talked about technical details, issues, missing features, etc. which of course are there. The technologies are still in their early days and scenarios for software factory use are still maturing. The learning curve is steep now, and that will definitely become easier in the coming years. However, it is absolutely possible to build some very good factories with the above described technologies. So, if you are interested in factories you better start looking at these technologies right away! 

Now that you know what to do, and what technologies to use for that, the next post in this series “How would you build it?” addresses how to put all these assets together in a process of assembling your factory.

Previous posts in the Factories 201 series:

What are they (concretely)? (Edward)

When would you build one? (Jezz)

To continue the factory 201 series let's discuss a next question: What do I tell my manager? Or maybe, how do we explain the benefits of software factories to the management to justify moving to a software factories approach.

Before we ask the management to move to a software factories approach we have to explain what the value proposition of a software factory is, and what is to be gained from a software factory approach. As we all know, most managers are not interested in technical details so we have to focus on the topics that are important for them! However, the value that we can get from this approach requires a change in the way we develop software today, and this requires a firm commitment from management to achieve success with this approach. Let’s have a look.

Please note that in this post I am using my own experiences from working for a System Integrator. These types of companies are building and using software factories in-house to deliver products for customers. Some of the topics I describe might be (slightly) different for other type of companies, for example for companies that build factories and sell them to their customers without using them in-house.

Value Proposition

The first step might be to tell them what a factory is, we can use the “What are they (concretely)?” post for that. The next step to get them really interested is to explain the benefits of software factories.

Increase Productivity

Probably the easiest benefit to explain is increase of productivity, decrease in cost. Factories automate parts of our development and design work and as a result of that the development team will be more productive during the development phase of the product, and that reduces development costs. But there is more!

Predictable, Consistent Results

Factories will also standardize the product that is developed and therefore we need less time for design and proof of concepts. Another benefit is the improved and consistent quality of the product that is built by the factory. After all, the assets for the factory are built by domain experts and remove the dependency on the very few highly skilled resources (sometimes referred to as ‘local heroes’) or over-resourced projects, to deliver the same high quality work.  A lot of companies are working on improving their software delivery process, reaching a certain CMMI levels and software factories can definitely help with that.

Re-use of: Skills, Resources, Value

Factories use abstractions to simplify the development of solutions. They encapsulate domain specialist experiences and knowledge into the assets behind these abstractions. As a result of that we don’t need to constantly staff each project with very best domain experts, since the assets themselves contain the required experience to solve the problem for us in the best way. Basically, the factory provides a collection of domain skills and experience we can use on each project. These assets represent the actual captured value of our skilled resources.

Furthermore, it’s possible we can re-sell these assets to other companies who may also find getting skilled resources a challenge.

Competitive Advantage

All of the above mentioned benefits are somewhat related to the product that is delivered from the factory, the resources we need for that and are mostly focused on the internal organization. Let’s have a look at some benefits that are recognized externally by investing in factories.

Software factories are hot and will probably become even hotter when they reach the larger community. This is obvious because the value proposition is so strong! Therefore companies can use the presence of software factories in marketing campaigns, and use them as a means to communicate their effectiveness and professionalism to their customers. They can display their innovation in this space and can use that in their recruitment activities to attract senior and skilled professionals.

Resource Attainment

Of course, in the end, our company can also leverage the factory approach to help solving its own resource issues. We can use the best resources to do the most challenging work and actually built the software factories. In this way we continuously challenge these resources, instead of making them participate in ‘boring’ project work and ordinary tools, to make it more likely they will stick with the company. The less experienced resources can become the factory users and be way more productive as they were before.

Commitments

Of course, after telling all of the above described benefits to our management they can't wait to adopt the software factories approach, but wait! There is more we have to tell them to make sure the adoption of software factories within our company has a fair chance to succeed.

Dedicated Resources

We already discussed the requirement for dedicated resources in a previous post (“When would you build one?”) but we cannot stress enough that you have to use the very best experienced, skilled architects and developers to build your factory. They have to be domain experts for the product the factory outputs but also have experience in software (tool) development in general.

However, we all know that the very best resources are the first resources to be pulled off a project and put on another. Since the value of factory assets (and therefore the factory as a whole) directly depend on these resources, it is critical they must not be removed from the factory development project. We all know, and so do the managers, that it’s our customers, market, and organizational culture who decide the allocation of our resources internally. To mitigate this we have to change these conditions to free ourselves from this powerful force to be able to keep these resources focused. The factory development project needs to be separated from the core business development projects, shipping ‘ordinary’ products, and requires a new organizational structure, cost structure, goals, culture and market to succeed. Management needs to support this.

Return on Investment

Return on investment is about profit and growth and thus more important than anything in the world for our management. We have seen the benefits that factories can bring us but before the management is willing to invest in software factories they probably want to know how fast they get their return on investment. After all, why would they want to invest in something that never pays back?

As we have already seen in our "When would you build one?" post, we probably need to build more than ~5 products with a software factory before it pays back so we better make sure we invest in a factory that is likely to be used more than once. We will discuss how and when this investment is returned in one of our future posts “How long will it take?”

Of course, the expectations on the ROI will be different than the established expectations for existing ordinary product development. So, there again, this calls for a separate organizational structure with its own values on what to expect from the product the factory delivers.

Not a Big-Bang Effort!

We have to make sure we keep the initial investment costs as low as possible. Therefore, trying to build a huge, fully featured factory from scratch in one attempt doesn't make sense, not only from a technical perspective but also from an investment perspective. We need to see a return on each iteration of the factory development.

To be successful, a progressive, iterative approach is preferred. The best thing you can do here is to start with small increments in automation of the product your factory creates. With each increment we identify the variable parts of the product and design those into the next iteration. This way, management can see an incremental return on every cycle.

To ensure your factory is applicable to more scenarios in its market, make sure management promote its use in many projects, even if the factory covers only a (relative) small part of the solution that is built. This way your factory can quickly identify many valuable scenarios of use, and support those in its evolving design.

Another option we have to minimize the initial investment costs is to re-use and customize an existing factory. For example, Microsoft patterns & practices released some (early examples of) software factories that are totally free and ready for use. It only takes us a few days to get familiar with these factories. After that we are ready to start using them in real life projects. Our project will benefit from the factory that we are using and in the meantime we will build experience with using software factories in our software development process. All of this with minimal investments but the experience we will gain from that will be invaluable for our future factory development activities. 

Measuring Success

One of the first questions we will have to answer when asking for an investment in factories is: "how do we measure the success of the factory?” In other words, we know the factory is supposed to increase our productivity, improve the quality and degree of standardization, but how do we notice and when will we see our first results? We have to make sure the management knows what to expect and therefore we have to specify this upfront.

For example, we can state the factory delivers a 5 % productivity increase during the development phase for projects larger than 2000 man hours, we can maybe save 80 hours during the design phase because we can use a standardized architecture documents and need less time for building a proof of concept. Further we have to specify when we can expect these results. For example, do we benefit from the factory in our first project or is it more likely this will happen in the second or even third project because the factory needs some improvements, people have to get familiar with it, etc. Of course, all of this depends on the state of the factory itself, the project, etc. but it is crucial that we use this kind of data to validate the effectiveness of the factory and the investment we are making in software factories.

Promote Use and Feedback

Factory development isn't something we do as a "stand alone" activity. After all, the factory is supposed to be re-used by a lot of projects and different project teams within our company. To make sure we don't suffer from the "not invented here" syndrome we have to make sure the factory authoring team, but also the management communicates openly about these initiatives. We have to tell our colleagues about our initiative, demo an early drop or even better, let them play with it and ask them to provide feedback, tips hints, etc.

Support Factory Adoption

Another thing we can do to increase the adoption of our factory is to help people use the factory on their project. Tell them a member of the factory authoring team is available at the start of their project to support them in using the factory. This will give them a head start, not only from the productivity and guidance they get from the factory itself but also from a domain expert that demonstrates them how to best use the factory in their situation. From experience we can tell this is something that is really appreciated by most projects. We can also ask projects that we supported to provide feedback to the management and tell them how the factory helped them in their project. This will definitely help us in marketing our factory approach within the company.

Summary

In my previous post "Software Factories: Where to start" I tried to list some resources that might be interesting for new comers in the software factory space. I have been thinking about factories and also participating in this space for some time now and I recently realized that, although there is a lot of buzz around software factories it is still very difficult to enter the world of software factories.

It might sound simple; install GAT and the DSL Tools and build your first factory but unfortunately, from what I experienced, it isn't that easy (it might be me of course!). I had (and still have) a lot of questions like for example: "What is a software factory", "When would you build one", "How would you build one", "What tools would you use" and many, many more.

It's therefore that  I decided to share some of my thinking's about factories to hopefully provide some answers to people that are about to join the software factories space. I am not a very formal guy so will try to write about these topics from a (factory) developer or (factory) architect perspective and keep it practical and understandable.

To be honest I am not sure I have the answers for all questions just yet but we will see where this leads too. Regular readers of this blog might have noticed that I discuss a lot of my software factory related work with Jezz. He did a lot of interesting posts on factories lately so hopefully he reads this post and will comment, or even better, add some content when he sees me struggling in my attempts (don't feel forced Jezz, but know that I am counting on you to help me out! ).

Ok, now we know the plan, let's make sure the title of this post makes sense and start with the first question:

What are they (concretely)?

If you were looking for a formal definition of what a software factory is, you would find one in the Software Factories” book written by Jack Greenfield and Keith Short. In this book, a software factory is defined as:

“A Software Factory is a software product line that configures extensible tools, processes and content using a software factory template based on a software factory schema to automate the development and maintenance of variants of an archetypical product by adapting, assembling and configuring framework-based components.”

Fair enough, but do we, as a developer, know what to build by just reading this definition? Not me. The patterns & practices team came up with a more pragmatic definition, that is in fact more accurate for the factories being built today:

“A software factory is a structured collection of related software assets. When a software factory is installed in a development environment, it helps architects and developers predictably and efficiently create high-quality instances of specific types of applications.”

But even this definition, does not really tell you what the factory itself is in concrete terms, at least not in enough detail to help you get started in building one. So we are going to use a much more simplified, and concrete definition that is more to the point from a developers perspective, describing what it actually is and how it physically works. Then from this, you will get a basic understanding of how the definitions above describe more precisely the moving parts of the factory.

[Please note, that this definition is being used here for the purposes of helping you understand the context of this series on building factories, and is not a formal and general definition of what a software factory is when it is finally shipped. The following definition only focuses on the concrete automation part of a factory. Inside the box of a real factory you will find other physical guidance assets like documentation, samples, reference implementations, training media etc. and together these make up a complete software factory.]

It’s a Tool!

First of all, a software factory is a software development tool. This tool is used to automate the assembly and/or configuration of a software solution that is addressing a well known and described, specific problem domain. It’s here where we see our first difference between a software factory and another development tool like for example Visual Studio.NET. A software factory is targeted at only a small problem domain while we can use Visual Studio.NET to basically build anything we want (of course within limits).

Configures and runs within Visual Studio

Although there is a difference between software factories and Visual Studio.NET (for the Microsoft platform) in the size of the problem domain they target, they are also very much related. This is because a software factory runs within Visual Studio.NET and basically configures this environment to target the development of a specific problem domain. Things like menus, windows, editors, layout etc.

For example, think of the Microsoft p&p Service Factory that configures Visual Studio to develop web services. This (early generation) software factory adds some menus, wizards and templates to the standard Visual Studio environment (Solution Explorer) to help developers build services.

Uses Abstraction

Another characteristic of a software factory is that it provides abstractions to simplify the problem domain it targets. An abstraction is something that simplifies the thing it describes and only focuses upon on only the important characteristics or variable parts of it.

In factories you can provide abstractions using several techniques like frameworks, specialized editors, wizards, models and diagrams.

Although providing abstractions is one of the most important requirements of a software factory, we haven’t seen a ‘publically available’ factory that uses all of these techniques - but I am sure we will see more in the future.

If you are interested in examples of abstractions that are used in factories, you might want to read this MSDN article about the EFx factory. The DSL that is used in this factory is a perfect example of the usage of abstraction.

Creates a Product

Another very important aspect of factories is its output. A software factory outputs a generated solution for the problem domain it targets. The product of a factory is what it outputs. A software factory doesn’t necessarily output source code. For example, it’s also possible that a factory generates some configuration files that are used to configure an existing framework, some documents, or whatever makes sense to the problem domain it targets. They can also just configure other products which solve a particular domain. One thing to keep in mind is, that is likely that the factory output needs some manual work or customization before it is ready to use.

Customization

There is also another type of customization that is very applicable to software factories. This is, the customization of the software factory itself instead of the output it generates. This type of customization leads to an iterative approach where we start with a small factory that iteratively increases. It is this type of customization that we use to address a new concern in the factory or, solve a defect in the factory product, or customize our assets to suit your development requirements and corporate standards.

Summary

So, here we are. I tried to explain what a software factory really is by describing some off its key characteristics without using common software factory terms like “product line”, “schema”, “variants”, etc. Because of that I might be missing some important stuff but I am sure I can make up for that in a way that make sense in future posts when I cover some other, software factory related topics.

For those of you who can’t wait for that, and like to read a little bit more about a software factories definition in more software factory related jargon, I can recommend this post from Jezz.

If there are any software factory related questions that you might have and want me to answer please let me know by commenting on this post. I cannot guarantee I have the answers but we can at least discuss about it and try to find the answer together.

To be continued…

As it looks like now, 2007 will be the year where software factories get a lot of exposure and attention. We have seen quite a lot of activity in this space in the second half of 2006 but there is definitely more to come! This also means that there will be a lot of new people entering the software factory space in the next coming months. For those of you who are not that experienced yet with software factories I decided to create a short list with some interesting links to existing software factories, forums, blogs, etc. Hopefully, this overview gives you a head start in entering the exiting world of software factories.  

Available software factories

If you are a practical guy (or girl) and like to get your hands dirty as soon as possible you might want to get yourself an existing software factory and start experimenting with it right away. In that case, check out the following offerings from Microsoft P&P. Clicking the links below will bring you to the MSDN landing page for each of the factories were you will find detailed information, links to the community and links to the download pages.

1. Web Service Software Factory
2. Web Client Software Factory
3. Smart Client Software Factory
4. Mobile Client Software Factory

You may also want to subscribe to the blogs of some of the P&P Product Managers: Don Smith, Tom Hollander and Eugenio Pace to keep you up to date with the latest news and future directives of the P&P factories. If you are interested in an indication of the "release dates" of the (future) P&P deliverables, check out their brand new "Upcoming Releases" page. 

Software Factories Tools

After experimenting with these existing factories for some time you might want to modify them (to make them better suite your requirements), extend them with some Domain Specific Languages or completely build your own factories from scratch. In that case you have to check out the technologies that drive the current wave of software factories.

First of all, we have the Guidance Automation Toolkit which you can use to create “packages” that can be installed in Visual Studio.NET and basically represents software factories as we see them today. You can download it from here and you can ask your questions about it (which you will definitely have ;)) here.

Second, we have the DSL Tools that you can use to create Domain Specific Languages that you can include in your software factories. You can download the DSL Tools from here and ask your questions about it here.

If you are interested in the combination of the Guidance Automation Toolkit and the DSL Tools you might want to read this article that describes some of the possible integration scenarios. (the library that is used in this article isn’t available yet, but will be soon!).

If you plan to start building your own factories, based on the above mentioned technologies, you also might be interested in the Clarius Software Factories Toolkit. This toolkit adds some extra features to the existing technologies to make it easier to develop your own factories. 

Background information, theory and concepts

Of course, there are also people who are really interested in the theory and the concepts behind the software factories strategy. If that’s you, go get yourself the Software Factories book written by Jack Greenfield and Keith Short. Once you have read this book you realize that we are still at the very early beginning with software factories and there is a lot of improvement to be made in the coming years. Some time ago I have written a post myself about some of the limitations in the first wave of software factories. As you can read there today’s software factories lack a model, schema and a lots of other stuff.

If you are interested in the jargon that is often used in the software factories space, check out this “Software Factories ABC” that I co-authored with Jezz. You better subscribe to his blog if you are interested in software factories because Jezz is an active factory blogger and has some really interesting posts about models, schema and tools. If you are still hungry for more information you can also check the software factories landing page on MSDN for more information.

So, we are done with the list. Of course there is a lot more interesting information about software factories but I am sure you will find them when you start your journey by experimenting with the currently available factories and reading the information I linked too in this post. Hopefully this list can be off any help to "software factories newbies" to get familiar with this very interesting technology as soon as possible.

Happy factories!

Update: also check out the Software Factories Swicki as the online search enigne for all factory related stuff!

For my current assignment, which is related to integrating existing and developing new Software Factories in our software development process, I decided to have a look at project GlidePath. According to the GlidePath website this project will deliver guidance on how to build software for Vista.  It delivers a web based repository of software factories that can loaded into the Visual Studio environment.  One of the very interesting features of GlidePath is the possibility to update the Software Factories over an RSS feed. It’s actually this feature that made me investigate GlidePath to see if we can use this or a similar approach in our own development process.

Once installed, you can use the Glidepath guidance package to create a Visual Studio solution structure. Instead of generating a full blown solution structure, like for example Service Factory does, Glidepath provides you with a rather empty solution.  At that moment the solution doesn’t reflect the actual product at all but rather provides you with the possibility to decide what factory to use in this solution structure (and thus what product to build?).  In this release of Glidepath you can only select one factory (MicrosISV factory) but I assume this will change in the future. Once I had added the MicroISV factory to my Glidepath solution I was surprised to see a lot of viewpoints popup in my solution structure (at this moment represented as .txt files).  From what I understood from Glidepath the idea behind these viewpoints is that you can select and configure them to specify how to build your product in the factory.  Glidepath will create new projects or enable packages (factories) in your current solution based on the viewpoints you configure for your product. Currently, Glidepath populates the MicroISV factory with viewpoints like “DataAccess”, “Deployment “, “Blogging”, “Legal”, etc.

Figure 1. Glidepath solution structure

After playing with Glidepath for a while I suddenly realized that I didn’t had a clue what kind of product I was actually building in my factory. I could see that my solution was populated with a project for “blogging” and project for “CSharp-WinForms-UX” because I selected these viewpoints for this solution.  However, from looking at the solution structure I wasn’t able to tell whether I was building a “blogging service”, a “blogging site” a “windows client for my blogging service” or whatever. Of course I understand this is related to this very early release of Glidepath and therefore lack of implemented packages, but I realized this was caused by the fact that the Glidepath factory doesn’t  have any notice (yet) about the product itself and therefore couldn’t tell me either what product I was building.

I was interested by this and tried to understand what this would mean for me if I had to implement (build a guidance package) one of the viewpoints for MicroISV. How would I know about the underlying product my viewpoint will be used on? Is it even necessary to have knowledge about that or can we generalize viewpoint implementations so they can be used on all products we are building? To be honest, I don’t have the answers for that at this moment.  My current thinking is that it would be helpful to have a formal description of the product I am implementing my viewpoint for but as said, I don’t have the answers yet.

A related question that interests me is: “what comes first?” Let’s say I was asked to build a factory for product X. Would I start with defining the viewpoints for my factory (security, deployment, etc.) or would I start by describing the basics of product X first? Or would I do this simultaneously?

Logically I would say, I first need a full picture of the product (product X) I am building the factory for.  Ideally, I would describe this product at a conceptual level (product model) in a way that makes this knowledge easily accessible and understandable for other people.  Once I know what I am talking about, and thus have understanding of the problem domain, I can decide what viewpoints are important, how to provide automated guidance for that, etc.

If we have captured the knowledge about the product we cannot only use this to author the factory but also to guide the factory user through the product at runtime. The factory can use the product model to communicate with the factory user in terms that make sense to the factory user and are related to the problem domain he is developing software for.

Currently Jezz and I are working on an experimental software factory authoring approach using a tool called the Factory Product Model (FPM).  I will not explain this approach or tool in detail in this post but if you are interested in more details you can read this, this, this and this post.

In this experiment, we are trying to validate whether a formal description (model) of the product the factory is building is a valid starting point for defining the factory and what it builds.

As far as I understand now, viewpoints will definitely play an important part in factory authoring and usage but what I am trying to find out is what impact they have and how they possibly relate to something like FPM.

I think specifying viewpoints is all about (some of) the requirements of the product we are building in the factory and therefore impact the type of guidance that is needed in the factory.

After all a viewpoint is what you get when you associate a work product, to an assets and a set of activities. Therefore viewpoints and work products are tightly related. It is just a question of which one do you define first?

I would be very interested in any ideas about this topic. What do you think? Do we need a product first approach (an FPM)? How does this relate to the Glidepath viewpoints approach I described? Do we need both? Do we need them in a specific order?

Let me know!

In a previous post I explained that Jezz and I are working together to prototype a different “approach” for authoring software factories.

In this experiment we are trying to validate this new approach based upon design ideas refined in the implementation of the EFX factory, and verifying that against the publicly available release of p&p Service Factory.

Basically, we think that we need to be able to let the factory author describe the ‘Product’ that is built by their factory at a higher abstraction level than we do now. We suspect that this approach will greatly help in improving the factory authoring experience and the factory user experience, which therefore simplifies software development as a whole.

Part of this experimental approach is describing the important concepts of the factory product in a so called “Factory Product Model (FPM)”. Below you can see a first attempt to model the Service Factory ‘Product’ in an experimental version of the FPM designer.

To make the example a little more realistic we introduced a few new concepts and made some assumptions.

For example we assumed it is possible to develop different type of services (entity services and process services) in Service Factory (there could well be others too). We did this to illustrate the concept of ‘Product Variants’. Which are basically one type of product the factory creates. Since a factory can create several variants of the product in its product line.

[Think of a ‘Product Variant’ as a very similar product but different in architectural definition, rather than something different in instance values of the concepts.]

Another assumption we made was that the difference between these services only reflects in the “service implementation” part. This is probably not completely true but we are only using these assumptions to simulate the situation where the factory enables us to build different family members of the service product line.

Figure 1. Factory Product Model for Service Factory 

The picture of the FPM for Service Factory shows us that we identified a “Product Variant” for each service type that is supported in (our imaginary) Service Factory. Also we identified the “System” product variant that makes it possible to group together any number of these services. We could have gone with one product variant for all types of services, but we wanted to define some that are different to validate the approach.

You can imagine then, that each product variant would be represented to the factory user as a separate solution template offered to them when they create a new solution using this factory. (In this case we would have 3 solution templates: ‘Entity Service’, ‘Process Service’ and ‘System’.

Looking at the concepts in the model, one interesting thing is the “Service Contract” concept; the (green) concept we are using, reflects that we will be using a custom Domain Specific Language for modelling Service Contracts in our factory. This custom DSL for modelling service contracts is responsible for modelling all the concepts that are defined within the “Service Contract” shape in the Service Factory FPM. This kind of model is called a Work Product Model (WPM), since it defines work products of the FPM. The child concepts of the ‘Service Contract’ concept will be refined in this WPM, and are represented here for logical structure.

The other concepts in the Service Factory FPM reflect common terms used in the Service domain that are therefore part of Service Factory. The concepts in this FPM will be represented to the factory user as a mechanism to “drive’” the factory. This view provides the user with a higher level abstraction than the current “Solution Explorer” view that is used in today’s factories. So instead of being confronted with all the technical details he can develop his services by using concepts that make sense to the service domain.

Figure 2. Example of the Service Factory Product Explorer

To make all of this happen, the next step would be mapping each concept in the FPM to physical solution structure, projects, and project items of Service Factory. Once that is complete, we define the assets (recipes, templates, designers, etc.) that will be used to create and configure these physical artefacts. Once configured correctly, the FPM is then used as a starting point for populating the factory schema for this factory. It already contains the definitions of all the work products, their assets.

Another idea we would like to validate in this experiment is the speculation that we can use the concepts modelled in the FPM to use as reusable parts in other factories. It would be great if we could find a way to just “query” this FPM (and/or factory schema) of a factory, and select the available concepts that we want to use to compose other factories from. You could imagine in this case that the ‘ServiceContract’ concept is self-contained enough to be reused in another factory (complete with DSL model and definitions).

In an earlier post I promised to share some more details about integrating DSL Tools with GAT. I am using some of the “techniques” in this post to experiment with integrating our Service Description Language with Service Factory.

As far as I know there are two possible approaches for integrating DSL Tools with GAT. In the first approach we simply execute GAT recipes from with the DSL Tools and pass the necessary arguments (we collect from the DSL model) to the recipe in the “execute call”. In the second approach we create “value providers” that are able to read the DSL model (selected shape, model root, etc.) and can be used directly in the recipe definition (“ValueProvider” tag) in GAT. It’s obvious that the second approach offers a more solid solution. For my experiment I tried both approaches. To be honest I don’t expect that I need my “experimental code” for too long. I can’t believe that Microsoft isn’t going to make something available to support this “out of the box”. This post of Mauro indicates that I might be rightJ.

Let’s have a look at the first approach and execute the “GenerateASMXDataContractFromSchemas” recipe from the “ASMXGuidancePackage” from the DSL message shape to generate the datatypes for all DataContracts modelled in the message shape of the DSL. The screenshot from this example can be found in this post.

Experimental hive

Before we start we have to make sure that the DSL isn’t using the “Exp hive”. GAT is registered in the “main hive” so we have to use the same hive both the DSL Tools and GAT for our integration experiment.

I did this by changing the .csproj files for the “DSL” project and the “DSLPackage” project.

In the “Dsl.csproj” file I deleted the “/rootsuffix Exp” from the <StartArguments> tag

                <StartArguments>/rootsuffix Exp /DesignTimeRun "..\..\..\Debugging\Debugging.sln"</StartArguments>

In the “DslPackage.csproj” I did the same action and also deleted “Exp” from the “<TargetRegistryRoot>”

         <TargetRegistryRoot>Software\Microsoft\VisualStudio\8.0Exp</TargetRegistryRoot>

Service Factory Guidance Packages

Of course we need to debug the new code for calling recipes in our DSL. To make sure the GAT packages are enabled for the “DSL debugging solution” I created a new solution based on the “ASMX Service” template of the “Web Service Software Factory” package. After that I changed the “debugging solution” for the DSL by replacing “Debugging\Debugging.sln” in the “<StartArguments>” tag in both project files with the correct path and name of the new solution based on the “ASMX Service” template.

After making the above described changes, the newly created Service Factory solution opens up as the debugging solution (in a new VS2005 instance) for our DSL. So, now we are ready to write some code to actually call the GAT recipes from the DSL.

Reference RecipeFramework dll’s

To make the code compile we need to reference the “Microsoft.Practices.RecipeFramework.dll” and the “Microsoft.Practices.RecipeFramework.Extension.dll” from the DslPackage project. The “Extension” dll is part of Service Factory and doesn’t have a strong name. To solve this issue we can open the “ASMX Guidance Package.sln” located in “C:\Program Files\Microsoft Service Factory” (default location), sign the extension project and compile it.

Code

To make the code work we need the following using statements:

using Microsoft.Practices.RecipeFramework.Services;

using Microsoft.Practices.RecipeFramework.Extensions.CommonHelpers;

I assume the code stated below is coded in the “OnMenu…” method for a custom command for one of the shapes in the DSL. Please not that the code example is just demo code!

In this first code snippet we can see that get a reference to the Visual Studio infrastructure. After that we get a reference to the “RecipeManager”. We will use this reference to instantiate the Service Factory Guidance Package. On my machine this is called “CustomASMXGuidancePackage”. This is because I already made some other modifications in the package.

The recipe that I am using in this example needs a reference to the Service Factory “DataType” project to execute correctly. We can use the “DteHelperEx” class located in “Microsoft.Practices.RecipeFramework.Extension.dll” to help with that. Unfortunately I wasn’t able to get a reference to this “DataType” project directly. This is probably because the project is located under the “Service” solution folder in the Service Factory solution structure. I didn’t take the time to investigate this any further so I took another approach for this demo code.

First I get a reference to the “Service” Solution folder which of type EnvDTE.Project. After that, I iterate over the ProjectItems in the “Service” solution folder to get a reference to the “DataTypes” project.

// Add DSL DataContract's to Service Factory DataType Project

Message message = ((MessageShape)this.SingleSelection).ModelElement as Message;

 

if (message != null)

{

    foreach (DataContract dataContract in message.DataContracts)

    {

        if (DteHelperEx.FindItemByName(dataTypeProject.ProjectItems, Path.GetFileName(dataContract.SchemaLocation), true) == null)

        {

            dataTypeProject.ProjectItems.AddFromFileCopy(dataContract.SchemaLocation);

        }

    }

 

}

                      

arguments.Add("DataContractsProject", dataTypeProject);

 

serviceFactory.Execute("GenerateASMXDataContractFromSchemas", arguments);

 

In the second code snippet we can see that we get a reference to the selected message shape in the model, iterate over all the data contracts in the message to add the XSD files that describe the datacontract to the “DataType” project (if not already added). The XSD files are referenced in a “SchemaLocation” property for the DataContract in the DSL.

After that we add the reference to the “DataType” project, needed as an argument for the recipe, to the Hashtable named “arguments”. You can find the necessary arguments for the recipe in the xml file that the recipe is defined in. In this case “GenerateASMXDataContractFromSchemas.xml” (located in “C:\Program Files\Microsoft Service Factory\Guidance Packages\ASMX Guidance Package\ASMX Guidance Package\Recipes\ASMX”). The last line of the code snippet finally executes the recipes on the instance of the guidance package.

Of course in this very simple example we are only sending one argument to a recipe that doesn’t have a “wizard form” attached to collect the values for the arguments. But the same approach can be used for recipes that do include wizard forms. However in that case it might not make sense to just display the values that are passed from the DSL to the recipe wizard form and ask the user to press the “Next” button on the wizard form.

For that situation it might be better to modify the recipe and/or write a value provider that can be used to read the DSL model values directly from the recipe. In a next post we will have a look how we can change a Service Factory recipe to make use of a custom build value provider that reads the DSL model.

[Update: added missing link]

In a previous post about “the next generation software factories” I wrote about the factory meta model and it’s role in providing a better user experience for the factory user. A more detailed explanation of the factory Meta model, or better, Factory Product Model (FPM) can be found in this post on Jezz’s blog. We can even see an experimental version of an authoring tool for the FPM in this post.

All of the above mentioned posts are using screenshots of the EFx Factory as an attempt to explain the Factory Product Model concept and it’s potential. However, because EFx isn’t publicly available at this moment we reckon that missing knowledge of EFx and its underlying domain might be debit to understanding the ideas behind the FPM.

So, to solve this issue we decided to take the first official drop of Service Factory from p&p and use that as a test case for discussing the Factory Product Model and possible future authoring tooling. Most people interested in software factories should at least be familiar with Service Factory as an early example so we think we can use this factory and the problem domain it is solving as a good example.

In the next couple of posts we will define the FPM for Service Factory by first using an experimental version of an FPM authoring tool. Along the way we will describe the important concepts of the FPM and demonstrate what Service Factory could have looked like if it leveraged a FPM. We will also describe the relationship of the Service Factory FPM to its underlying physical solution structure and how all of this related to a fictional Factory Schema for this factory.

We will not only use this series of posts to share information to help peoples understanding of these concepts, but also to validate and refine our current thinking in this space so any feedback is highly appreciated!

Let’s start. The first thing to do when authoring a factory is to make sure we fully understand the problem domain the factory is supposed to solve. For the Service Factory, this domain is already well defined and is reflected in the Service Factory solution structure. We will use the following concepts from Service Factory in our discussions:

• Service Contract
• Service Interface
• Service Operation
• Message Type
• Data Type
• Service Implementation
• Business Entities
• Business Logic
• Data Access Logic

Some of the above Service Factory concepts are directly realised in the Visual Studio solution structure. Others, for example Service Operations, are “hidden” somewhere in a C# code file and can only be viewed by opening the physical file.

We will use the above list of concepts together with their physical representation (projects, source files) in Service Factory to model a FPM for Service Factory. We will then use this higher level abstraction of the Service Factory to see if we can further improve the factory user experience by driving Service Factory from this abstraction.

Stay tuned…

Service Description Language meets Service Factory

I have been working for quite some time now with the DSL Tools and spend the last couple of months on Service Factory and related to that a little bit on the Guidance Automation Toolkit. Now, I decided to integrate the two and see how easy it is to make customizations in Service Factory and call GAT recipes from the DSL Tools.

Both our DSL (Service Description Language) and Service Factory cover the same problem domain of “designing services” so this seems to be a perfect fit ;)

I decide to start simple and added two new custom commands to the message shape in our DSL. As you can see below we now have the opportunity to call the “Generate DataTypes” and “Generate MessageType” recipe from the message shape.

My first concern was to just make it work so I didn’t spend any time yet on optimizing the Service Factory recipes to make them match better the DSL. I reckon this is necessary to make the user experience of the DSL integrated in Service Factory better.

If we have a look at (some of) the properties of the DataContract (picture below) that is modelled in the message, we can see that the DataContract is defined in the “RestaurantData.Xsd”.

After selecting the “Service Factory: Generate DataType” command on the message shape we can see that the XSD that describes the DataContract is added to the Service Factory “datatypes” project and that a datatype class is generated out of the XSD file.

Another thing we can do now is generate a Message Type for the message that is modelled. In the picture below we can see that we can execute the “Service Factory: Generate Message Type” command and that we can send the necessary data from the message in the model to the recipe in the Service Factory package.

As said, the Service Factory recipes and wizard need some changes to make them fit better with the DSL. For example in this case we don’t need the “Response Class Name” textbox on the wizard because we are starting the recipe from one message in the model. To solve this we can of course execute this recipe from the Service Operation in our DSL or customize the Service Factory recipes.

In a future post I will share some more technical details on how to call GAT recipes from a DSL, how share data between the DSL and GAT, etc.

For me it was good to see that it is possible to combine DSL Tools and GAT and that is relatively easy to make changes in the Service Factory package (not shown yet).

To be continued...

In the last couple of months I have seen the enormous power of these first examples of software factories but also saw some of pain points exhibited by them. I decided to write some them down also describe some of the (possible) features of the next generation software factories. At some points I am using the EFx factory and my discussions with Jezz as a reference for this post.

Where are we today

After the introduction of the Software Factories initiative and the publication of the “Software Factories” book it took quite some time before the first examples of Software Factories became available. Apparently the Software Factories concept was so totally new that people needed this time to get familiar to the concept and possible advantages.

Even now, when looking at the first examples of Software Factories, we can see that most of these examples are in their infancy. One of the reasons for this is the fact that not all Software Factories concepts, like for example Software Factory Schema and Software Factory Template, are not as well defined and understood as we would like them to be. As a result, there is very limited tool support for authoring factories.

Tools like the Guidance Automation Toolkit (GAT) and Domain Specific Languages Toolkit (DSL) have proven to be successful in these early examples of factories but their learning curve is steep and these assets only offer authoring of some of the work products which make up a factory, they offer not support of the factory overall as a cohesive development environment.

Early examples

When we have a close look at some of the characteristics of the first wave of factories from the patterns & practices team, for example ‘The Service Factory’, we can see that they are primarily based upon GAT recipes only. These ‘super’ guidance packages make extensive use of related recipes to create source artefacts in a solution. The factories are technically driven to automate the common tasks of architects and developers that would normally have to be hand crafted.

Whilst the recipes of these factories encapsulate valuable knowledge and experience of the solution domain, they act only as a means to write source code which has to then be subsequently tailored towards a specific solution. These factories make little or no use of modelling to describe the problem domain and offer little abstraction. Hence these factories still require expert knowledge of the solution domain in order to exercise them effectively.

As a result, the factory addresses a problem domain from a technical, Visual Studio related point of view instead of a more conceptual perspective that simplifies the problem domain itself.

Because of this tight coupling of these factories with the physical artefacts of a solution we cannot easily differentiate in views between the various types of factory users. This means all users experience the Factory in exactly the same way.

Another issue in the current wave of Software Factories is a missing, formalized way of describing software factories. An often heard concept in this space is ‘Software Factory Schema’ but till now nobody seems to know exactly how this is implemented. As a consequence the Software Factory Schema isn’t supported or integrated in any form of tooling in these early factories, and consequently we are still faced with a solution structure based approach in the early examples of these factories.

Software Factory User Experience

As stated previously, the user experience of today’s software factories is governed by the view of a solution in terms of its physical artefacts and therefore has a rather technical solution based focus. Let’s have a look at the two views that currently represent the factory in today’s examples.

Solution Structure view

The first view that can be identified is the “Solution Structure” view. This is a view we are all familiar with and has been around for a long time (as seen in the Solution Explorer tool window). It provides an overview of the Visual Studio solution structure and shows us all the different artefacts, like projects and source files that our software solution is composed of. Developers and architects familiar with Visual Studio know how to use this view and nobody really complains about it. Simply because it just represents the way we are developing software today.

Besides an overview of the solution file, project files, assembly references and code files it also reflects the architectural layering of the software component. It even provides us with build information, the naming of the individual components and the way the software will most likely be deployed. Basically, this view tells us a lot more about the physical infrastructure of the solution, the .NET framework and the build engine than it tells us about the purpose and concepts of the software we are developing.

In actual fact, the solution structure is often tailored by developers to describe the physical deployment view of a solution and the logical structure view of it at the same time. Often the result is a compromise of both these views, neither fully satisfying either, and hence every development team has opposing opinions on a best representation of these views.

This view is always the same for all project members. We have no way to filter the content of this view. The architect, the developer and the tester are all faced with the same view of all artefacts to represent their different concerns. Most of the time this isn’t necessary at all and it even makes software development more complex than needed.

Guidance Navigator view

The second, so called, “Guidance Navigator” view, is a new view driven by necessity by the Software Factory initiative. A first example of this view is available in the factory examples from the Microsoft patterns & practises team. This “Navigator” view provides us with a clear overview of all the activities (recipes) that can be executed in the solution.

Although we can also start these activities from the context menus on the individual items in the solution view it’s the “Guidance Navigator view” that summarises all the recipes and makes them easier to discover in one place. Another benefit of the “Guidance Navigator” view is that actions are tagged with additional help and background information to improve first time factory users understanding and improve the user experience. This is one example of the support for the “guidance in context” pillar of software factories.

However, an important aspect that is missing in this current implementation of the “Guidance Navigator view” is the lack of support for pre- and post conditions for activities. We need these conditions to control when activities become valid next steps for the factory user who is navigating the factory. Without this capability the user is still required to know which recipes to run in which order and with no way of telling the state of a completed activity as a whole. This view also currently lacks the capability to filter activities based upon the role of the factory user.

What we can learn from this “Guidance Navigator view” is that we can dramatically improve the usability of our software factory by just representing the same information in a different way. The “Navigator view” is a good start and it does provide us with a much better view on the factory activities but, we are not there yet! It still doesn’t provide us with a conceptual view on the underlying problem domain of the factory and thus the software we are developing. 

Software Factory Schema

One of the things we are currently missing a way to describe our software factory in an understandable and predictable way. The suggested means for this is the ‘Factory Schema’. This schema is often referred to as the “recipe” for the software component we are developing in the factory. It lists all the ingredients like, source files, models, frameworks and describes how to combine them to build the software component. It also describes the different views we have of the individual parts and the factory as a whole.

Although this description is quite clear it isn’t so clear what pieces of information should actually be part of the software factory schema. Even if we are able to identify all individual parts of the software factory schema most of us don’t know in what form we have to describe this to make it usable. Till now there is no formal, publicly available specification of the software factory schema that we can use to write our own schema. One of the things we do know is that a well described and implemented software factory schema might be the key to success for future software factory development and usage. The Software Factory Schema offers very important benefits for the software factory user.  

Let’s have a closer look at the software factory schema and try to describe the most important parts of it. The first concept in the schema is the “viewpoint”. Basically a viewpoint is a related set of activities, artefacts (work products), etc. that have a relationship. We can use the viewpoint to specify what elements of the software factory will be valuable and thus visible for a specific type of software factory user (stakeholder). It’s the viewpoint concept that gives us the opportunity to specify how different types of factory users will experience the factory. This might be based on the role of the factory user which makes role another concept in the software factory schema. Viewpoints can be related to each other by creating mappings between viewpoints.

Now we know viewpoints contain a set of activities we also know that activity is another important concept within the software factory schema.

An activity can be defined as a set of actions that we have execute in order to produce a work product or artefacts. Activities can be grouped together in a set of related activities.

A third concept in the Software Factory Schema is the workproduct. A workproduct can be best described as “anything” the software factory produces. It can be anything from one single xml file to a combination of code files or models of part thereof. Workproducts can be hierarchical grouped together. By grouping together workproducts we can define additional abstraction levels on top of the low level workproducts like for example source files. We will later see how we can use these abstractions to boost the usability of software factories. Activities use assets on their turn to produce workproducts. Think of assets as code generators, frameworks, domain specific languages, etc. that are used to create the workproducts. Workproducts have state that will change over time from, for example, created, partially ready to finished, by executing one or more activities.

(Have a look at this post from Jezz for more detailed information about the Software Factory Schema.)

Software Factory MetaModel

Another important concept in the Software Factories area is the Software Factory MetaModel. The Software Factory meta model helps us describe the software factory at a higher abstraction level. As we have seen earlier, today’s factories are mostly described and exposed by using the capabilities and terms available in the Visual Studio.NET infrastructure (Solutions, projects and files). Important concepts in our factory currently are reflected only in project names, names of source files, and layout of the projects and solutions etc. A concept that the factory defines which may include several source files cannot be easily represented in physical solution structure, and also many of the physical source artefacts created by Visual Studio tools have little meaning in the schema of the factory.

Therefore we need a Software Factory meta model – a model of the concepts of the factory that the user understands in terms of the problem domain. We can use this meta model to define abstractions that exist in the problem domain that the software factory deals with. In other words the meta model provides us with a mechanism we can use to describe our factory at a more conceptual level. We can define the so called software factory concepts in this meta model and relate them to each other. To make this concept a little more clear let’s have a look at the following example.

Think about a software factory for developing services. Thinking about the “service domain” we might identify concepts like “Service Interface”, “Service Operation” and “Message Contract”. When building a software factory for this domain by using the current tooling we will see that these concepts only reflect in documentation, project names and files names. The factory is driven by executing recipes that eventually add source files to solutions structures. There is no way to navigate through the factory by, for example, iterate through all the “Service Operations” that are available in the solution that is built with the factory. Simply because for the factory itself and Visual Studio.NET this concept doesn’t exist.

All that the first wave of factories have knowledge about at this moment is a set of files grouped together in a Visual Studio solution together with a set of recipes.

Figure 1 shows us how a service, built the Microsoft Service Factory, is represented to the factory user. Later on, we will see how we can change this representation by leveraging the factory meta model.

Figure 1. Representation of a service built in Service Factory.

The presence of a Software Factory meta model would enable us to describe the concepts that exist within the problem domain and thus are valid for the software factory. A populated meta model for the above described factory would provide us with a capability to navigate through the factory by using concepts like “Service Interface” and “Service Operations” instead of navigating through the projects and underlying files in the Visual Studio solution.

To demonstrate the power of the software factory meta model we will use EFx factory which is the first factory that leverages a factory meta model. EFx is an architectural guidance factory that is an offering of Microsoft Services. We will use EFx factory to introduce the “software factory view” that is built on top of the factory meta model.

Software factory view

The software factory view is the third view we can identify in the software factory area. In this view, we are displaying the factory concepts, defined in the factory meta model, to the factory user. The factory user isn’t confronted with physical artefacts like solution, projects and source files. The result is a view that allows the user to construct the products of the factory in terms of the problem domain. As we can see in figure 2 which represents the software factory view for EFx factory we are now dealing with concepts like Applications, Subsystems, Business Entities, etc.

Figure 2. Software factory view in on top of the EFx factory meta model.

The hierarchy in this view doesn’t reflect the underlying file system or solution explorer structure at all. Instead, we are looking at the conceptual structure of the software we are writing. We can execute our recipes on these concepts, and have them relate to one or more physical artefacts in the solution such as source files, diagrams and other models that extend the factory schema.

The advantages of this view are numerous. First of all, this view simplifies the problem we are dealing with during our development activities. Developers don’t need knowledge of all the underlying details like project structures, files, deployment scenarios anymore. All they are faced with are the conceptual pieces of the software they are developing together with the activities, implemented as recipes that are related to the concepts. They can execute activities to produce the associated workproducts without worrying where to put it in the Visual Studio solution structure or store it on the file system. With this software factory view in place we can even change the complete underlying Visual Studio solution structure, to for example meet specific architectural requirements and deployment scenarios, without developers even knowing about it.

Another advantage of this view is that we can leverage the viewpoints we defined in our software factory schema to hide specific concepts from different types of software factory users. This means we can target the factory user experience based on their role to better meet the requirements of the different types of users. We can even define a viewpoint that only displays the concepts that make sense to business analysts or end users so we can discuss the software development process with them.

A view like this requires that the factory maintain the state of the Meta Model, rather than creating physical artefacts using recipes. Instead recipes are used to configure the meta model, and physical artefacts are then generated at a later stage from the meta model.

Summarized we can say that we can use the software factory view to represent the factory concepts described in the meta model to improve the user experience of the factory.

In some future posts I will elaborate a little more about some of the above described concepts. In the meantime check out Jezz blog for a lot more interesting information on software factories and EFx factories in particular.

It’s been a long time since I wrote about the never ending story of the small demo Domain Specific Language we are building. In the meantime I received a few mails asking me for the status and Steven Kelly even mentions in this post that I probably gave up on the DSL Tools.

Well…, as we wrote in our recently published article (still only available for subscribers) we have plans to make it downloadable as soon as the V1 of the Microsoft DSL Tools hits the streets. To prepare a little for that I decided it was about time to install the latest June CTP of the DSL Tools and try if I was able to get this DSL working in the new DSL Tools.

I used the migration tool that comes with the June CTP to migrate my old domain model and designer definitions into the completely new format. Although this migration went quite smooth, I decided to start from scratch again, just to get familiar with this new release of the DSL Tools.

To be honest, that took some time. I considered myself an “experienced” DSL Tools user so decided it wasn’t necessary to read the (limited) documentation. I was wrong! Because of that I noticed far too late that there is a “DSL Details” window that helps you set some (important) properties for the domain model and designer. This was one of the reasons I didn’t get things working. Once I found the DSL Details window (View/Other Windows from the menu) things became much easier.

Another issue that took some time was that during debugging my DSL the shapes didn’t appear after dropping them on the design surface. They were added to the underlying model and did appear in the “Explorer” window. After some time I found out that I had to add my domain classes as “Element Merge Directives” of the root (in my DSL “Service”).

The good thing about the new DSL Tools is that you (almost) don’t need to manually edit the underlying XML anymore. I only needed that to make my compartment shapes work. Once you understand the concepts, the “DSL Explorer” makes it relatively easy to build a domain model and designer and it looks *nice*!

As far as I am concerned, the new June CTP is a very nice upgrade of the DSL Tools.

Now I only have to migrate all my custom code (if still needed), add some validations and include the WSDL generation code and I am done J.

Hopefully there will be no breaking changes for V1!

The article about Domain Specific Languages I wrote together with Christian Weyer for CoDe magazine is now online. It’s called “Introducing Domain Specific Languages” and briefly describes the Software Factories initiatives before it focuses on Domain Specific Languages in general and our DSL for the remainder of the article.

At this moment only a piece of the full article is available for non-subscribers. This means you either have to subscribe to CoDe magazine or save the link and get back in 2 months when the full article is available for all site visitors of the CoDe magazine.

 Hope you like it!

I was playing a little bit further with the DSL Tools and tried to add some icons to the custom commands (menus) I created. With the help of Jezz I managed to get this working.

First of all we tried to use custom bmp images in the commands. Unfortunately there appears to be a bug somewhere in the VS shell that prevented us from using custom bmp images. One option to work around this is use Guidance Automation Toolkit for adding the custom commands, we decided to follow another approach and use one of the icons from the office resources. If you can live with this limitation (approx. 7000 icons), you might be interested in the information below.

First you need to decide which icon you want to use in the menu. Therefore, you need the so called FaceID of the icon that you want to use. There are several tools or Office add-ins that can be downloaded from the internet to help you determine the FaceID of an icon (I used this one).

For this example we will use FaceID 2497.

Now we know the FaceID of the icon we want to use, we can open the “C:\Program Files\Visual Studio 2005 SDK\2005.10\VisualStudioIntegration\Common\Inc\office10\msobtnid.h” file in VS.NET.

This file contains all ID’s for the Office icons. For this example, we search the file for FaceID 2497. We will find a line that looks like:

#define msotcidExploreWebPage          2497   //(43,25) Label: 'E&xplore Web Page'

From this line we get the declaration of the FaceID. In this case this is “msotcidExploreWebPage”

In VS.NET we now open our DSL solution and open the “Designer\CtcComponents\GeneratedCmd.dslddt” file.

Somewhere after the include of the “GeneratedCmd.dslddi” file we can add the following line.

#define OI_GENERATEDIAGRAM guidOfficeIcon:msotcidExploreWebPage

Of course you can change the “OI_GENERATEDIAGRAM” in whatever you want. Change the declaration (“msotcidExploreWebPage”) to the appropriate value for your icon, leave the “guidOfficeIcon” in!

Next, we open the “Designer\CtcComponents\CustomCmd.ctc” file and find our customer command. This line should look something like this (in your case, of course with other name and text)

guidCmdSet:cmdIdGenerateModel, guidMenu:grpidContextMain, 0x0200, OI_NOID,  BUTTON,            DIS_DEF,       "Generate Model from WSDL";

We can change the “OI_NOID” into “OI_GENERATEDIAGRAM” (your value here)

Last, we have to open the “Designer\Shell\Package.dslddt” file.

To make sure VS.NET resets its menu cache after the first build we need to increment the second integer in this line:

[ProvideMenuResource(1000, 15)]

Build and run the solution and the icon will appear in your command.

Doesn’t this look pretty? 

My friend Jezz started a new series about the work he is doing on building a Toolkit for developing enterprise distributed applications and services. I know some off the details of his work and this is quite interesting. Jezz is just like my pasionate abouth DSL, GAT, and all related stuff. Let’s see what he has to say in his coming posts...

Finally! A little more information is available of the DSL Tools modelling API! In this post George Mathew provide us with (some) details on the improvements the DSL Tools team made in the modelling API.

As you might have noticed from my previous posts, we have a quite a lot of custom code in our Domain Specific Language that uses this modelling API. Therefore changes in the modelling API will reflect in some additional work for us also. Hopefully not to much!

According to George, creating new Domain Classes can now be done by using the ‘new’ keyword. Isn’t that great? (and obvious J). The good new is that also the “old” mechanism is supported in the new API. Hmmm, I think I will definitely upgrade my code to use the ‘new’.

Great work, waiting for more news George!

It’s been a long time since my last post. I have been busy lately working on a few pretty cool things. Unfortunately cannot talk about most of the things yet. But, there’s one thing I can talk about and that is the “ever lasting story” of our Domain Specific Language for designing service contracts.

We still plan to make it available for download as soon as the RTM of the DSL Tools is available. In the meantime we are adding some nice features that hopefully improve the usability of the Domain Specific Language. Of course, we have to make some (hopefully not to many) changes to make our custom code work against the changes API of the RTM of the DSL Tools.

For now, let me show you some images of some of the features and/or improvements we have been working on. Just to show you this language isn’t dead yet!

First of all, as you can see in the image below, we included some nice images in the ToolBox. Not that difficult but I like the Toolbox a little better now.

Further we added some custom menus. One of them to fire the “WSDL generation” another one to fire the “generating model out of WSDL file” option.

Below you can see an example of two custom menus that are available when right clicking a MessageShape in our model. When pressing the “Select Message XSD location” (description will probably change!) you get a FileDialog box to select an XSD file that describes the Message. After you selected the XSD, the “schemalocation” property of the MessageShape in the model gets populated with the reference to the XSD.

A next thing you can do is right click the MessageShape again and this time you can select the “Select Message from XSD” option. As you can see in the image below, this action results in a new window that pops up and that is populated with all the message definitions in the XSD that you selected in the previous action. You can also click on the messages to see the DataContracts that are referenced in the message.

After selecting a message the MessageShape in the model gets populated with the name of the selected message and the DataContracts belonging to the message are also added to the MessageShape. Maybe no rocket science, but it hopefully makes the Language a little easier to use.

To be continued...

The new November DSL Tools CTP comes with a totally new Validation Framework. This framework can be used to write constraints for domain classes in your domain model. After trying it out I must admit that the framework makes writing constraints very easy! All validation errors popup in the “Error List” windows in VS.NET, all of this is very cool!

However, I was a bit confused. I assumed that whenever a constraint was violated on a domain class in my model this was reflected in the “IsValid” property of the ModelElement and the “Store” that represents my model. None of this is true! While the Error List window shows some constraints violations the “IsValid” of the “Store” (and the ModelElement) returns “true”.

Because I need this information in a piece of custom code in the DSL I had to look a little further. I noticed that it is possible execute the validation myself from within the custom code. Have a look at the following code snippet where I am using the “ValidationControler” class to execute the validations. (don’t forget: using Microsoft.VisualStudio.Modeling.Validation;)

ValidationController controller = new ValidationController();

bool isValid = controller.Validate(this.Store,ValidationCategory.Save);

This code lives in the “ServiceDescriptionDiagram” class in the DSL. From there I have a reference to the “Store” that I can use in the “Validate” method. The methode has some other overloads that makes it, for example, possible to validate an ModelElement. The good thing is that this piece of code actually returns the right value (based on the violated constraints in my model). In the code above the value of “isValid” is false whenever a constraint is violated.

I am not sure (yet) why the IsValid property on the “Store”, “ModelElement” and “ShapeElement” doesn’t reflect the current state of any violated validation. It might have something to do with the “ValidationCategory” that is part of the validation process. This category can be specified on a constraint to tell the validation framework when a constraint should be validated. Current options are “Menu”, “Save”, “Open” and “Custom”.  It looks like no information about a model or class being valid is stored in the underlying datastore of the model. I assume there is a very “valid” reason for this!

For continuing our DSL, this isn’t a problem at all. We can use the “ValidationController” at any place in our DSL to decide if the model is valid enough to start generating artifacts.

To be continued...

Some days ago I received an email of somebody asking me if I am (still) using T4 templates within the DSL we are building. After reading some of my latest post he got the idea that a lot of the features we are building aren’t implemented in Text templates (T4). He asked me if there is a special reason for that and if I think the T4 templates are useful enough for writing (parts of) your DSL.

Actually it is true that a lot of the things I am currently working on is implemented in pure C# code and is hooked into the diagram itself by using techniques described in earlier posts. For me this works just great.

The thing I like about integrating as many features as possible directly in the diagram itself is that it requires no user actions (selecting menu items) to “execute the feature”. For example, in our DSL you don’t have to select a menu item to translate a Service that is modelled in our Designer into WSDL. This is done on the fly whenever a change is made in the designer and the diagram is still a valid representation of a Service. All of this is done by events that get triggered “automatically” through the designer. Another option was to implement the WSDL generation into T4 templates and let the user trigger the execution of the T4 template through a menu item.

It’s not that I don’t like writing T4 templates! Actually I do like the templates. I have to admit that there are some improvements possible in the “writing experiences” of these templates but I am sure this will improve in future builds of the DSL Tools. I can imagine that we will use the T4 templates for some other artifacts that get generated out of our domain model. To be honest, the WSDL generation for our DSL is besides the “pure C#” version also available in a T4 template. Translating the “pure C#” version into a T4 template (and vice versa) was done in a few minutes.

So, to answer the question in the email that starts me writing this post. A lot of the functionality in our DSL is built in “pure C#” directly hooked into the diagram, but that doesn’t mean that I don’t like T4 templates or think they are not useful! In fact, they will be part of our DSL. I think you have to find out for yourself what features and/or customizations you need in your DSL and what can be best implemented in pure C# and what can be best implemented in T4 templates.

I noticed that George Mathew (team member DSL Tools) wrote one of his first posts on his blog about new features in the Text Templates (T4) of the DSL Tools. Maybe he (or other team members) has some thoughts or better explanation on when to use the T4 templates and when to implement things directly into the designer or diagram?

In an earlier post I mentioned I am using the “OnTransactionCommiting” on the Diagram class to implement some logic (generating WSDL) for the DSL we are writing. My remark at that point was that the event is triggered a lot and therefore might not be the best place to implement time consuming logic. At that point I didn’t pay a lot of attention to the “TransactionCommitEventArgs” parameter of the event. After some further investigation I found out that the “Transaction” (that can be reached through the TransactionCommitedEventArgs) has a logical name that “indicates” what is happening in the diagram and what (user) action triggered this event.

For example, when changing the value of a property named “WSDLDescription” in the diagram, the name of the Transaction is “Set WSDLDescription”, when first loading a diagram the name of the Transaction is “LoadDocData”, etc. The code snippet below gives you an idea how to limit unnecessary (custom) code executions that you might implement in the “OnTransactionCommiting”.

public override void OnTransactionCommitting(TransactionCommitEventArgs e)
{
   base.OnTransactionCommitting(e);

   if ( e.Transaction.Name.ToUpper() == "SET WSDLDESCRIPTION")
   {

      //only execute in case of a change in the WSDLDescription property
      DoStuff();
   }
}

This actually means that I have much more control than I initially thought and therefore makes the “OnTransactionCommiting” a usefull event to implement some other logic too.

To be continued…

One of the things I *really* want to implement for our DSL is “round tripping”. This means in my case, (at least) generate a “Service Description diagram” out of a valid WSDL file and visa versa.

Some weeks ago I already did an attempt in this direction but eventually I ran into problems with the serializer. As you can read in this post on the DSL forum the relations between concept got lost when trying to serialize (save to disk) the in-memory representation of the generated diagram. Probably this is caused by either a mistake on my side or a bug in the DSL Tools. Some days ago while playing with the propagation rules I found a way to make “round tripping” work in our DSL. Let me explain a little further.

I added a new property to the “Service” concept in the DSL, called “WsdlDescription”. This property represents a reference to a WSDL file describing the service. I added a new propagation rule that fires whenever the “WsdlDescription” property of the Service concept changes. This propagation rule, on its turn, fires a piece of code that parses the WSDL file (referenced by the “WsdlDescription” property) and generates a diagram out of this. With this code in place I can generate a diagram out of a WSDL file by simply creating a new empty diagram. To be more precise, adding a new item to the Visual Studio solution based on the “ServiceDescription” (name of the DSL) template. This results in an (valid) empty diagram. I can set the “WsdlDescription” property (service concept) in the empty diagram to reference a valid WSDL file and the magic happens, the diagram is generated! The advantage of this approach, compared to creating an empty diagram from scratch by using custom code, is that we can rely on the DSL Tools infra structure to properly initialize the diagram.

Of course you can find another “hook” to implement your round tripping code. This depends on the DSL you are building. For this moment, in our DSL, the propagation rule on this property does the trick.

Let’s have a look at some code. Below you see the code for the propagation rule that is defined for the “Service” concept in the model. First we try to cast the input parameter of the “ElementAttributeChanged” method to a “Service”. If that succeeds and the rule is fired because of a change in the “WsdlDescription” property we can continue. To create any new concepts in the diagram we need the “Store”. This can be seen as the in-  memory representation of the diagram. We can get this by using the “Store” property on the “Service”. Then we create a “ServiceInterface and ServiceOperation and register both elements under the Service by using the “ServiceArtifacts” relation. (To better understand the model of this DSL, have a look at the, slightly outdated, image of the domain model of the DSL). Finally we add the “ServiceOperation” as an “Operation” under the “ServiceInterface” and we are done. We now created two model elements and a relation between them in code. All displayed perfectly in the diagram.

[RuleOn(typeof(OurCompany.Design.ServiceDescription.DomainModel.Service),
                                                     FireTime = TimeToFire.TopLevelCommit)]
    public sealed class DomainModelServicePropertiesChangesRule : ChangeRule
    {
        public override void ElementAttributeChanged(ElementAttributeChangedEventArgs e)
        {
            
            // cast input parameter to Service element
            OurCompany.Design.ServiceDescription.DomainModel.Service service =
                                 e.ModelElement as OurCompany.Design.ServiceDescription.DomainModel.Service;
            if (service != null)
            {

                // only execute it when the rules is fired because of a change in the WsdlDescription property.
                if (e.MetaAttribute.Id ==
                        OurCompany.Design.ServiceDescription.DomainModel.Service.WsdlDescriptionMetaAttributeGuid)
                {
                    
                    // create new ServiceInterface. Use the store of the service
                    ServiceInterface serviceInterface =                                                                 ServiceInterface.CreateServiceInterface(service.Store);
                    serviceInterface.Name = "GeneratedInterface";
                    
                    // add ServiceInterface as an ServiceArtifact of the service
                    service.ServiceArtifacts.Add(serviceInterface);

                    // create new ServiceOperation. Use the store of the service
                    ServiceOperation serviceOperation = ServiceOperation.CreateServiceOperation(service.Store);
                    serviceOperation.Name = "GeneratedServiceOperation";

                    // add ServiceOperation as an ServiceArtifact of the service
                    service.ServiceArtifacts.Add(serviceOperation);

                    // add the relation from serviceInterface to serviceOperation
                    serviceInterface.ServiceOperation.Add(serviceOperation);
                }
            }


        }

    }

In the above code snippet all of the diagram generating is hard coded, this is, of course, just for this example. In my situation the hard coded stuff is replaced by some calls to helper functions that parse the WSDL and translate this back in a diagram.

To make sure the rules gets executed on changes in the Service concept we need to facilitate that it can be “picked up” by the “rule engine”. This is done by reflection; therefore we need the following code.

    // This class is reflected on to load rules dynamically.
    // Add a variable for each custom rule.
    internal static partial class GeneratedMetaModelTypes
    {
        internal static Type DomainModelServicePropertiesChangesRule =
                                    typeof(DomainModelServicePropertiesChangesRule);
    }

Now the diagram is generated out of the WSDL, we have to make sure the WSDL stays in synch with any changes that might be made in the diagram. It’s my intention to let this synchronisation magic happen automatically, so without any user actions (pressing menu options, etc.). To experiment with this I added the “synchronization logic” to the “OnTransactionCommiting” event of the diagram. As you can see in the code snippet below this is just a matter of overriding the “OnTransactionCommiting” method in a partial “ServiceDescriptionDiagram” class.

public partial class ServiceDescriptionDiagram
{
   public override void OnTransactionCommitting(TransactionCommitEventArgs e)
   {
      base.OnTransactionCommitting(e);

      //validate the diagram and execute synch code


   }

}

Although this works perfect I am not absolutely sure if this is the right place to implement this kind of logic. I noticed that the code gets fired a lot. Within this method I first validate the diagram to make sure there is enough information in the diagram to generate a valid WSDL file. If so, the generation (synchronisation) starts. I haven’t experienced any performance issues with this approach but maybe it is better to only synch when the diagram gets saved. I have to do some more investigation on this area.

To be continued….

The MessageHeader concept in our DSL has an attribute “SchemaLocation” (the same is true for Message and DataContract). This property is a reference to the XSD description of the MessageHeader. Another property of the Message concept is the “Namespace” property. The value of this property is used in the WSDL that is generated out of the Domain model of our DSL. In an attempt to make our DSL as user friendly as possible I decided to set the “Namespace” property automatically as soon as the “SchemaLocation” is populated.

To make this possible I wrote a propagation rule (one of the customization features of the DSL Tools) to make this happen.

Below you can see the code that is needed for a propagation rule. I stored this code in a separate .cs file in a folder called “custom” that I created in the designer project. The rule inherits from “ChangeRule”. The implementation isn’t that difficult (note: just demo code)

Because the code that actually retrieves the “Namespace” out of the XSD file is stored in another assembly that isn’t part of the DSL assemblies, I had to add the assembly to the GAC to make it work.

To make sure the rule gets fired we need to add a variable to the “GeneratedMetaModelTypes” class. This class is reflected on to dynamically load rules. The variable is the reference to the new propagation rule described above. The following code snippet will do the trick.

Internal static partial class GeneratedMetaModelTypes
{
        internal static Type DomainModelMessageHeaderPropertiesChangesRule =
                                    typeof(DomainModelMessageHeaderPropertiesChangesRule);
}

As you can see in the image below (I know this image doesn’t prove a lot!) the “Namespace” property gets set after we populate the “SchemaLocation” property of the MessageHeader with a valid value.

To be continued...

Because we made some good progress with building the core functionality of our DSL I decided to spend some time on the customization options of the DSL Toolkit.

I tried some of the customizations described in this document that can be found on the DSL Tools forum and also implemented the “ToolTipText” for all of the shapes and connectors by using the code that is described in this post of Pedro Silva.

Another thing that I found was necessary was changing the color scheme of the Shapes. I decided to use some RGB colors and found out that using an “RGB color code” in the designer.dlsdd file isn't an option. It wasn't too hard to found out how this can be achieved by writing some lines of custom code. As you can see in the code snippet below it is just a matter of overriding the FillColor property in a partial class for the Shape.

public partial class ServiceOperationShape
{

    public override Color FillColor
    {
        get
        {
            return Color.FromArgb(132, 150, 172); 
        }
            
    }

}

The Message concept in our DSL is holding an embedding relation for the DataContract and MessageHeader concept. Therefore our MessageShape is a compartment shape this means for these two compartments there isn’t a possibility to override the FillColor property in a partial class.

We had to find another way to changes the color. It turns out that compartments can be reached in custom code by using the “compartmentDescriptions” collection on the parent shape. As you can see in the code snippet below I used the “OnInitialized” of the MessageShape (parent) to set the “TitleFillColor” property of the two compartments in the MessageShape to an RGB color code. (Maybe there is a better place to set the color but this does work. Another option would be the “ OnShapeInserted“)

public partial class MessageShape

{

      public override void OnInitialized()

      {

            ...

MessageShape.compartmentDescriptions[0].TitleFillColor = Color.FromArgb(132,150,173);

MessageShape.compartmentDescriptions[1].TitleFillColor = Color.FromArgb(189,162,123);

      }

}

As you can see in the image below the color of the compartments are changed (compared to the colors in some previous posts). Also a “TooltipText” is displayed when moving the mouse over the connector shape. The tooltipText is composed out of the information of the two shapes that are connected by the connector.

To be continued...

Great news! The DSL Team just released a guide that describes how to customize the features and capabilities of the DSL Toolkit (September CTP). Thanks guys!

This guide describes for example how to use custom code to create: “transparent text decorators”, straight lines, shadow and gradients, container object, etc. All techniques that can dramatically improve the modelling experience of your DSL.

A lot of other interesting features are covered in this document including the explaining code samples. Of course al of this is subject to change in the next releases of the Toolkit. But if you don’t like that, don’t use CTP products and wait for the V1 release!

Anyone interested in this guide, go get it following this link to the DSL Toolkit forum.

Have fun and expect a lot of the features described in this guide in the DSL we are building.

It’s been some time since I wrote about the progress of the DSL we are working on. Not because there is no progress anymore but mostly because I am very busy with my regular work so I simply have no time left to write a post. It’s still my intention to have the DSL fully operational by the time V1 of the DSL Toolkit is available and so far I don’t see any reasons why this wouldn’t be possible.

I am now using the September CTP of the DSL Toolkit. Unfortunately the migration of the whole project took some more time than expected. I followed all steps of the (perfect) migration guide but still I got an error when trying to use the designer of my Language in the debug instance of Visual Studio.NET. It took me some time and a little help of the DSL Team to find out that my designer definition file contained some “shape definitions” that I wasn’t actually using anymore. These shapes were now part of a compartment shape in my language but I had forgotten to delete the definitions from the file. The May CTP build didn’t have any problems with this. The very cryptic error message (“package …. Has failed to load properly”) wasn’t very helpful for finding the problem. So solving this issue took some time. The good thing of course is to see that more and more validations are built in the Toolkit. I have good faith that the final release will provide more descriptive (helpful) error messages for these new validationsJ.

What are we working on now? Currently we are working on the “generating WSDL part”. This WSDL will be based on the service description that is modelled by using the DSL. For me this basically this comes down to writing some T4 templates to mix the “service description” (modelled in the language) with some pure C# code that Christian wrote to generate the WSDL. This work is almost done. From there we can do basically everything we want with code generation.

There isn’t much information available about writing T4 templates. Anybody interesting in writing these templates might find some help in this blog post of Gareth Jones (DSL Team) about the T4 syntax. A little information about how to navigate over your DSL model can be found here on the Domain Specific Language Tools forum.

Some directives that I needed within the templates:

To include files (create some sort of a function library):

<#@ include file="filename.extension" #>

To reference an assembly:

<#@ Assembly Name="System.Web.Services.dll" #>

To import a namespace:

<#@ import namespace="System.Web.Services.Description" #>

A lot more information about writing templates will be made available in the V1 release of the Toolkit. By then all the API that can be used for writing templates will be stable and no longer subject to change.

To be continued…

Currently I am working on code generation for the DSL we are building. The DSL Toolkit comes with some sort of a framework that can be used to build templates for generating artifacts (based on the model). In this release of the toolkit however, writing templates is a poor experience (just my opinion J). The templates are displayed as “text files” within the Visual Studio.NET 2005 IDE without any syntax coloring. This makes large templates very hard to read. As you can see in the image (small part of the template) below it is very hard to make a distinction between the C# code that is needed for generation the artifact and the C# snippets that are part of the artifact and thus part of the template. The image below is only a simple and small part of the templates so imagine a large template.

The good news is that the DSL team is working on (or at least thinking of) providing a richer template writing experience. Hopefully this will make it in the next CTP release of the DSL Toolkit! I definitely I can’t wait for a better coding experience!

Enough complaining. As you can see in the two images below, it is possible to generate some code based on a model. The template above also shows that navigation the model isn't too hard either (foreach ServiceArtifact in this.Service.ServiceArtifacts). The image shows the C# code that is generated based on the definition of the “service interface” in the DSL designer. No rocket science but it works. Of course, more code can be generated but this is just an example.

One of the artifacts that I will generate (based on the model) will be an xml representation of the modelled concepts and their relations. I think that might become handy if I want to use another code generator engine that isn’t based on the DSL Toolkit Templates and thus might not integrated very well with the DSL model. Hopefully this gives me some more flexibility when it comes to generating the artifacts in the future.

To be continued…

It’s time for another update on the progress of our DSL. Christian and I have been working on the underlying model of our Domain Specific Language we are building. We made some changes to make the model better reflect the “service design” domain.

For anyone interested in some background information on this topic (service design), go read this article of Christian. It explains some fundamental concepts that might help you understand the domain model a little better.

Please note that the model below is still work in progress. It is very likely that it will change in the near future. Some of the concepts and relations are based on restrictions in the current release of the DSL Toolkit and might not reflect the perfect “designing services” world. Also getting the code generation to work properly might force some changes in the underlying model. For now let’s have a quick look at the model and briefly discuss a few concepts. (I am sure in the end a more detailed explanation of the complete model will follow)

If you have seen the previous version of the model you notice the new “DataContract” concept in this version of the model. The “DataContract” describes the data that we need to exchange. In our language the “DataContract” is a reference to an XSD description of the data, therefore the “DataContract” concept has a name, “schemalocation” and a “namespace property”.

The “Message” concept (previous “RequestMessage” and “ResponseMessage”) has an embedding relation (solid line in the model) to “DataContract”. The “Message” concept reflects the messages we need to exchange the data that we defined in the “DataContract”. A message can hold one or more “DataContracts”. Messages can be of the type “Inbound” or “Outbound”.

The “ServiceOperation” concept in the model reflects the definition of a  serviceoperation of the service. The “ServiceOperation” has a “MessageExchange” property that can be set to either “OneWay” or “RequestResponse”. Based on this property the serviceoperation will have a reference (dotted line in the model) to one or two “Messages”.

Another interesting (?) design decision in the model is the embedding relation from the “Service” concept to the “Message” concept. We want to explicitly display the “Message” and “DataContract” concepts on another level than the other “ServiceArtifact” concepts. As you can see in the picture below of the “ServiceDescription Explorer” (part of the designer of this language), there are two child notes below the “Service” (CustomerService) concept. The “Messages” node as the root for the messages and datacontracts and the “ServiceArtifacts” node acting as the root for all other items.

Ideally we would have displayed the messages and datacontracts under the appropriate serviceoperation in the “ServiceDescription Explorer”, but to get that to work we needed some changes in the relations between the concepts. In our case this would result in concepts participating in more than one embedding relation which is not allowed in the current release of the DSL Toolkit. This will probably change in the next upcoming CTP build!?

On the left of the “ServiceDescription Explorer” you see an example of these three concepts used in the designer based on this early preview of the model. Here we modelled a services operation (GetCustomerDetails) referencing one inbound message (GetCustomerDetailsRequest) and one outbound message (GetCustomerDetailResponse). Both messages hold one or more datacontracts in the message.

Hopefully this clears some things up for the “DataContract”, “Message” and “ServiceOperation” concepts. More information on the other concepts in another post. 

To be continued…

It’s been a while since I wrote about the progress of my DSL experiment. Why does it take so long? Is it so hard to build a DSL? Well, actually sometimes I think it is. It takes some time to really understand what is going on in all the files that are part of the DSL solution. Most of this is because of the technology preview status of the toolkit and therefore the limited documentation that is available. In the current release some of the files have to be maintained by hand which can only be done if you have a good understanding of the xml structure of the files. Most of this manual work will disappear in the next release of the toolkit. Luckily there is an active forum and the DSL Toolkit team that is willing to help!

But, let’s continue on the status. Three weeks ago Christian Weyer of Thinktecture contacted me to let me know he liked the work I am doing in my DSL experiment. As you can read in this post on his blog Christian is thinking about using DSL’s for modelling service interfaces and data contracts for some time. We decided to discuss this “service modelling” domain via email and to join forces and build this language together.

So, the last few weeks we discussed the artifacts that can be identified within the service domain and how to model them in our DSL. As a result some of the names of the concepts that I modelled in an early version of the language have changed to make them more descriptive and some new concepts are introduced. Currently I am making the last changes in the model. After that we have to finish the designer for the language and finally we can do some code generation. In another post I will show a next version of the model and hopefully explain some of the ideas behind it.

Hopes this clears thinks up. The experiment is not dead. Expect to hear of it a lot more in the coming days, weeks, months.

To be continued…

Now that I am in the process of building my own DSL for modelling services I started thinking about the usage of DSL’s a little more. Suddenly I see all kind of opportunities for new DSL’s but also some issues so I thought why not share them with you and ask for your opinion.

Thinking about the “service DSL” I am experimenting with a little further, a possible next step would be to build a language that models the cross cutting concerns for services.

Suppose we have all (or at least some J) our crosscutting concerns modelled in a DSL and we are able to simply configure a service with the crosscutting concerns by using the DSL designer and “drop” the crosscutting concerns on the service area. (For now, let’s assume it is possible to attach the crosscutting concern to the service by only using a configuration file without the need to make any changes in code.)

A language like this would make it relatively easy to generate the necessary config file that actually attaches the crosscutting concerns to the services that are “hosted” in the WSE pipeline, Indigo Windows Communication Foundation or some other piece of infrastructure.

Although being able to attach crosscutting concerns to services would be great for creating the initial configuration file the next step is of course to also maintain these “settings” for production software. Unfortunately there are some burdens on our (my) way to make this happen. For example, the current release of the DSL tools only support “one-way” generation. This makes it impossible to generate a model based on an existing configuration file. I assume this will change in the future. Is it Microsoft? Similar functionality is already available in the VS.NET class designer.

A second issue might be the hosting of the DSL. Until now the DSL designers can only be hosted within the Visual Studio.Net IDE. If Microsoft isn’t going to change this in the future, IT Pro, responsible for the running the services in production, have to use the Visual Studio.Net environment to maintain their configuration files. Forcing the IT Pro department to use Visual Studio.NET for maintaining and supporting software (configuration) might sound strange but this might become reality sooner or later. If we believe Microsoft IT Pro will start using VS.Net to create and maintain their logical data centre models in the near future. On the other hand it might be good to have some sort of a light weight hosting environment for DSL’s.

A third thing that comes to mind is the “size of the DSL” and the possibility to share concepts between DSL’s. One of the goals of a DSL is to target a certain problem domain. In my example, we have a DSL for modelling services and one for attaching crosscutting concerns to services. Both languages share common issues and therefore in theory can be integrated in one language. But wait, this would mean that our IT Pro department that is only supposed to use the “crosscutting concern part” of the language also has the possibility to use the “modelling services” part of the language. Modelling (and building) services is definitely no responsibility of IT Pro and thus integrating the two different features (modelling and crosscutting concerns) into one language might seem reasonable from an domain perspective but doesn’t reflect the responsibilities of the IT Pro employee very well.

An obvious option is to keep the two features in separate languages. In this case however we have the “service concept” that is part of both languages. Of course we can model (and maintain) this concept in both languages but is that what we want?! The ideal situation would be to have the possibility to “import” concepts from one DSL into another DSL and have the capability to use some sort of access control on the details of a concept. In this example we could have one DSL targeting “services” but make sure the IT Pro department can only use the “service concept” for attaching crosscutting concerns to services and cannot view (and use) all details that the developer can use to model the service.

Of course the above issues are described by using the “service” domain I am experimenting with but I assume they can be translated to a more general case.

What do you think, should we “worry” about issues as “one way” generating, DSL hosting environments and sharing concepts between DSL’s or just focus on things we can do with the current DSL technology?

Just a quick update:

I received two emails of people asking me if my little DSL experiment has stopped. The answer is NO!

Because of some positive feedback I decided to spend more time on the design of the language. At first, the model was mostly meant to help me understand the DSL toolkit, now I’ am actually trying to build a usable language.  

That’s why it is a bit quite on the topic, just because I don’t think it is very interesting to write about every small change in the language design.

In my opinion the experiment is getting more interesting and definitely more useful.

To be continued…

Although my model isn’t quit finished yet, I decided to see how my language behaves in the Visual Studio experimental build environment. You can enter this environment by executing “Start without Debugging” from the VS.Net “Debug” menu in the VS.Net environment you are designing your language in.

Within the experimental build environment I created a new “serviceoperation” class (GetCustomerDetails). After that I created a new “responsemessage” (GetCustomerDetailsResponse) class. And last, a relation between the “serviceoperation” and the “responsemessage”. The model for this looked something like this (actually the little collapse picture in the right corners wasn’t there at first).

To be honest I wasn’t quite happy whit the “modelling experience”. Besides the poor colours of the classes (which can be easily changed) I wasn’t sure if this is the best way of using the designer for this part of the language. Therefore I had another look at the possibilities and found out about “compartment shapes”. A compartment shape can be defined as a shape (class) that can host another shape (class).

Here is what I did; I changed my original “serviceoperation” class into a compartment shape that is capable of “hosting” a “reguestmessage” shape and an “reponsemessage” shape. To create a compartment shape you have to make some changes in the designer xml file. Not that straight forward but I’ am sure this will change in the next release of the toolkit!?

As you can see in the picture below the “serviceoperation” class now hold to “placeholders”. One for the “requestmessage” class and one for the “responsemessage” class.

By right clicking the response message placeholder you actually get a menu option to create the “responsemessage” class (see below). Till now my messages classes in my language only have a “name” attribute. I have to test what happens if my messages classes contains a little more attributes. I’ am not sure how this is represented in within the designer and how easy it is to view or change example the message attributes. I assume only the name if the message will be displayed in the serviceoperation class and by double clicking you can change the attributes but a little more effort is needed on this. (In the image below I right clicked the RequestMessage placeholder to add an new “requestmessage“ class).

For now, imagine yourself designing (and generating the code) your services with a language like this. I think it’s pretty cool!

To be continued…

I found some spare cycles to continue my DSL experiment.

I wasn’t happy with the initial version of my domain model, so I decided to make some changes in it. Although the model isn’t the most important part of the experiment I figured that it should at least be possible to compile it to continue the experiment.

After making some changes in the domain model, I ran into several problems when I tried to get the designer file in synch with the domain model. As a result executing the “translate template” step caused a lot of errors that I was not able to solve.

Let me try to explain what happened. Suppose we have the following model:

When I used the “modelisoft add in” to get the designer file (.dsldd) in synch with the model I received the following warning:

Warning : in MelCollection of shapemap for 'ResponseMessage', Concept

> 'ServiceOperation' has a role 'ResponseMessage' but it is not

Aggregate

I figured that I made some fundamental errors in my model but couldn’t find anything about it in the accompanying DSL Tools walkthrough documents. After some time I decided to contact Alan Cameron Wills who is a member of the DSL Tools team and is very active on the DSL Tools forum. Within half an hour he responded and solved my problem. Thanks Alan!

He told me that in the current version of the DSL Tools there’s the following restriction:

 All concepts of all shapes that appear on the diagram must be embeddings of the root class.

In practise this means that models should be very flat. What I had to do is create an embedding relation (solid line in the diagram below) from “service” (root of the model) to “RequestMessage” and “ResponseMessage”. After that I had to change the right-hand roles “Min” attribute to 1 and change the “Accepts” attribute of the left-hand roles to “All”

(changes are within red lines)

Maybe a bit strange but it works!

After applying the suggested solution my model looks like:

Note: the model isn’t finished yet. This picture is just to explain this restriction!

Alan also mentioned that the future release of the DSL Tools will have a much better “modelling experience”, till then we have to use the above described solution to get around this restriction.

After installing the DSL toolkit it is relatively easy to create and maintain the “domain model file” by drag and drop items on the designer that comes with the toolkit. After that, however, you have to get the “designer model file” in synch with the “domain model file”. This file is used to generate C# code that makes it possible to use your language in the experimental build version of Visual Studio.Net.

Getting this designer file in synch, really is a pain in the ass! It is a lot of work and for people not very familiar with the designer file structure (yet) it’s very likely to make mistakes with it. I assume Microsoft didn't have the time to solve this issue for this release of the DSL Toolkit but I hope they will solve it for the final release!?

Fortunately I found this Visual Studio add-in that makes it real easy to keep the two files in synch. Modelisoft, thank you for making my life a little less painful!

According to this posting of Daniel Cazzulino, Visual Studio Add-ins are dead. This might be true but this add-in is definitely alive and kicking!

Found some time to continue with my little experiment today.

After executing the installation of the earlier mentioned toolkits I was ready to perform my first steps in this totally new world. Installation all went fine, so that a good thing. One of the things I started with was reading the so called walkthrough documents (at this moment four docs are available). Unfortunately these documents are the only documentation that I can find for the DSL Toolkit. This is probably because of the CTP status of the toolkit but I assume this will get better in time. Although the walkthrough docs provide some guidance I think it is a bit poor. The docs mainly tell you which button to “click” but do not provide you with real background information about the DSL concepts. Definitely some room for improvement in this area!

My first experiences with building the example stated in the first “walkthrough” document, were a struggle. Most of this was because I didn’t take the time to execute all of the steps stated in the document. (I guess this is just a normal “men” thing to first start trying it yourself and only read the manual when you f.cked up). Second time I followed the docs very carefully but I was left with some errors. Luckily I found this posting on “”Xenopz Blog” that helped me with some remaining issues and that couldn’t be resolved by just reading the manual.

After some time I decided to start with the “Blank Language” template and start building my own DSL. This template provide you with two classes (“ConceptA” and “ConceptB”) and some relationships between them.

I started building my own DSL by simply renaming the two classes in the model (this is suggested in the walkthrough documents). One of the things that I find a little inconvenient (other words came up in mind first) in this is that after you changed the name of the classes and relationships in the designer you also have to perform some “find and replace” actions in one of the (many) xml files to make the designer for your language work. 

After some time this is what the first version of my model looks like.

During the creation I didn’t make any attempt to execute the “translate template” step or build the solution. Thinking back I’ am not sure if this is the smartest thing I could do.

My next step will be executing the “template translation” (to get the designer definition file in sink with the above domain model) and build and debug the language.

Haven’t tried this with this model but I expect some troubles with it. Getting the designer definition file in sink includes making manual changes in the (complicated) xml file. More on that later.

For, now Just a short explanation of the model (language):

At the root of the language I created the “service” class. Just below that you can see the “servicecomponent” class. I figured that all the other service components (interface, operation, etc.) might share some common needs (properties) in the future. Therefore all service components inherit from the “servicecomponent” class. 

The relation between the “servicecomponent” and the “servicecomponent” class is there to facilitate creating relationships between the several service components. Further you can see the “serviceAdpater” that acts as the class that couples the “serviceinterface” and “serviceimplementation” class. The “serviceinterface” class groups one ore more “serviceoperation” classes and is referenced by a “transportinterface” class. Finally the “serviceoperation” class references two “message” class to support the modelling of an input and output request for a service operation.

All still very limited, but it gave me a nice introduction in some of the possibilities of the DSL Toolkit. The above language can now be used in a so called “experimental build” environment to actually try the language and model a service. I haven’t tried that yet, so I’ am not sure what to expect!

When writing this post I suddenly realize that the two references from the “service operation” to the message class might not make sense. Maybe it is better to extend the message class with an attribute that indicates the type of the message (input or output). By using an enumeration for this I can make sure only the correct types can be selected. After that I can delete one of the referenced message classes and change the remaining reference to accept one or two messages.

To be continued…

Inspired by the several sessions of Tech-Ed Europe I decided to spend some effort on Domain Specific Languages (DSL) and Guidance Automation Toolkit (GAT). Although I have done some reading on Software Factories the area of building a Software Factory is quite new for me. Therefore I thought it would be interesting (for me and my current employer) to see the capabilities of these new technologies that are first members of Microsofts Software Factories initiative. To make it a little more appealing than just creating some sort of “Hello World” example I decided to try to combine it with some of my “day to day” work (which is architecting systems). Because I am definitely no expert in DSL’s I also decided to share some of my thoughts and issues to hopefully get some feedback to push me in the right directions.

Here is the idea: build a Domain Specific Language (DSL) to model “services”, perform some code generation and use GAT to provide some context-specific menus and templates within Visual Studio.Net IDE to speed up development based on the DSL.

Doesn’t sound too difficult so let’s give it a try. One of the things I do know about building a Software Factory is that you have to define what (reusable) parts can be identified within the domain. You also have to make sure the complete implementation details for all of the parts you are trying to model in a DSL and automate with GAT are available (and proven!).

For now, let’s assume these conditions are met. I will describe some of the parts I think are members of the service domain. Because my focus is on getting familiar with DSL and GAT, I will not describe all parts (and details) within the domain. As soon as I am totally up to speed with DSL and GAT (and still find it interesting) I will complete the model to make it a little more useful.

One of the most obvious parts that can be identified within this domain is the actual “service” itself. When translating this to a DSL this can be the “root” of the DSL. The service is composed out of other parts that I will call “service components” (it’s just a name).

Some of the service components I identified for this example:

• Service Interface; operations that are exposed by the service (service contract).
• Service Operation; definition of a service operation.
• Transport Interface; the representation of the service interface targeted at a specific technology (web service, enterprise service, remoting.)
• Message; definition of the messages exposed by service operation
• Service implementation; the actual implementation of the business logic
• Service Adapter; layer between the transport interface and the service implementation.

The above service components are all in their most basic form and they might not even cover the whole service domain. For example, it is very likely that in reality the message contains a data definition that is implemented as a data class based on an XSD definition. For now (this example) I will forget about al of these details. In this example a message is just represented by a class with a name that can be used as either input or output for a service operation.

I think, I will soon find out what parts (or details) I am missing here when I start modelling my DSL. During the modelling phase I will add only the absolute necessary properties and relationships.

The next step will be to prepare the environment by installing the DSL Toolkit and the Visual Studio 2005 SDK for Beta 2 and than we are ready to start experimenting with DSL.

To be continued…