Factories 201 Series - Building Software Factories

• What are they (concretely)? (Edward)
• When would you build one? (Jezz)
• What do I tell my Manager? (Edward)
• How long will it take? (Jezz)
• What would you build? (Jezz)
• With what would you build it? (Edward)
• How would you build it? (Jezz)
• What should you strive for? (Edward)
• What are the challenges? (Jezz)

Post authors: Edward Bakker and Jezz Santos.

Well, we have finally come to the end of this 201 series on ‘Building Software Factories’.

First time arriving here? - we have listed the entire series above – please have a look through those posts before ending it below.

If you are arriving here after reading (hopefully all) the posts in the series, we wanted to say, good on you, and thanks for sticking it out with us.

Series Wrap-Up

Well, we have spent quite a bit of time and thought in putting this series together for you. The motivation behind this effort has been that we’ve recognised that there is little practical information helping ordinary professional developers on getting started with building and understanding software factories. We have had quite a head start on this and wanted to share our knowledge and experiences with you and the community to promote the uptake of building factories, which in turn should promote the adoption of software factories and the industrialisation of software in general.

This series was created in a format that asks a logical sequence of questions that you might have when trying to figure out how to build software factories today. We have covered many such issues as they arose and shared much of the combined knowledge and experiences built up over the last few years in this space. We will be sharing more resources on this in future posts – so stay tuned to our blogs.

We hope that you can take the guidance represented in this series forward as a starting point for developing your own software factories further. We recognise openly and frankly in the series that this is not a trivial undertaking today, but hopefully we have shown you that it is very possible and how to successfully approach it. Hopefully we have also directed you to what’s most important, where you should direct your energies, and how to build factories practically and realistically.

Your Feedback

One of the primary objectives of this series was to connect with many more of you in the community and share ideas and innovations about everyone’s efforts at building factories today.

We have received a small number of comments to this series so far, which we have been keen to discuss at length, and hope we have done a good job at answering the questions appropriately. Hopefully, there are many more of you out there building factories, or at least considering building them, and so we hope to hear more from all of you on that. Please drop us a comment on the appropriate post and we will work hard to get you an answer you will find useful.

If we are doing a good job please let us know. If we are doing a poor job and you don’t like the guidance we are giving please let us know that also. This should shape any future guidance we give the community in this space, and at least we can start further discussions with those of you in this space. Hopefully, we will also get a feel for just how many of you are out there are in this space.

The Final Say

There should now be a little more information out there to help you build a software factory right now on the Microsoft platform with the current toolsets. This series was created exactly for that purpose to share with you the experiences we have had in building factories and give you insights into the issues we have faced on this road already.

As more and more people jump into this space we expect to see from the community more detailed, formal guidance on this subject, so that the community as a whole can learn from previous experiences and elevate the adoption and the application of software factories to real world projects.

After all we think that the act of building a software factory should leverage the tenants that software factories promote themselves, i.e.

“Package domain expertise and guidance in a form that others can re-use and manipulate to create solutions using simplified descriptions of the problem at hand.”

Hopefully we have done a good job of that for you, and hopefully you will consider the experiences we have shared in this series in your factory building efforts.

Come see us...

We are proud to announce that we will be presenting a session on ‘Real World: Building Software Factories’ at Tech-Ed 2007 in June this year.

The 300 level session will contain material from this series and demos and examples of other factories to explain the concepts. We would love to meet up with you there and answer your factory building questions.

Many Thanks, Jezz Santos and Edward Bakker

Previous posts in the Factory 201 series:

• What are they (concretely)? (Edward)
• When would you build one? (Jezz)
• What do I tell my Manager? (Edward)
• How long will it take? (Jezz)
• What would you build? (Jezz)
• With what would you build it? (Edward)
• How would you build it? (Jezz)

After reading all of our previous posts (don’t tell me you skipped one!) we know what automated assets to build, which tools and technologies to use and how to build a factory. The final step here will focus on the quality attributes we have to keep in mind during the design and development of our factory in order to make it a stunning success! In other words, “What should we strive for?” when building our factories?

Abstraction

In a previous post “With what would you build it?” we already mentioned the power of using abstractions in our factory. Let’s dig in this a little further and see why abstraction is definitely the top thing we should strive for!

Increased Productivity

One of the advantages of using abstractions in our software factory is significantly increased productivity. We can think of an abstraction as a simplified representation of a specific problem. In fact, we are probably using abstractions in our job as a developer or architect every day. (i.e. 4G programming languages, class diagrams, API’s, class libraries, frameworks, DSL’s etc.)

Think about it, for example, if we build a helper class to perform a specific task (say: logging to a file) we are actually using abstraction. We create a class and some methods that simplify the details of logging, file management, rolling-over, naming, log details and formats etc. Once we have the helper class in place we can just re-use the helper class using some basic parameters without focussing on the details of how it works anymore. We solved our problem more simply by creating the helper class.

Now, let’s elaborate on this a little further. Suppose we only use this helper class in certain circumstances of a particular type of solution, let’s say only for certain components of a particular application. In this case, we don’t use or need all the possible parameters of the helper class (because we only use it in a certain way in this context), and let’s say we identify 5 different cases of use, each with different parameter values. Now we can build another higher-level helper class that wraps the original helper class, and defines only one parameter which describes one of our 5 cases – perhaps an enumeration. Our new helper class is now an abstraction of the older helper class – because it simplified its usage in this particular context. Typically, this new level of helper class becomes part of a framework for those types of applications, because we have made certain types of assumptions about how the helper class is to be used.

[To clarify this, using abstraction for simplifying a problem is something we see all time. For example, take ADO.NET that hides a lot of the low level details that are needed to communicate with a SQL Server database. On top of that we have Enterprise Library that adds another abstraction level to make data access even more straight forward. In the future, ADO.NET vNext and the Entity Framework will do an another attempt to raise the level of abstraction for “data programming”]

As we continue to build up the levels of abstraction, you can see that we can solve more specific problems at less level of detail, and also by using a higher level description or language. If we can get close to a point where we can provide these abstractions in terms of the specific problem we are solving, we are making it real easy to program our application (based on minimal information). This is where productivity is gained.

Abstraction Roadmap

When developing a solution you can think of many abstractions, when you add modelling and automation to this picture the possibilities for abstraction build upon each other vastly:

Platform API’s → Application API’s →Helper Functions →Class Libraries → Frameworks → Domain Models → Software Factories

Each of the abstractions denoted above increases the level of abstraction of the problem you are solving, and in each case also narrows the domain of the solution you are providing for – this is the simplification part. I am sure it’s obvious to you that using a domain model to configure an application is far quicker than writing programming API’s to describe the same thing.

The other key point here is that I can configure my application in terms of high level things that make sense to that application and not in the low level details of the API’s. This means that those using the domain model now don’t need to be programmers they can have higher level skill-sets like solution design, requirements gathering and interpretation etc. By raising the level of abstractions we are moving from a physical design (platform API’s) to a logical design (domain model) that exposes less details and is understandable for a much wider audience.

Automated Guidance

Another very important aspect we have to keep in mind during the development of our factory is the level of automated guidance we expose to our factory users. In our post “What would you build?” we have already discussed the different types of automation assets that we might want to use in our factory but we have make sure we use them in their most effective way.

Remove Mundane Tasks

A top quality attribute (from a developer’s perspective) for using our factory would be to provide automated assets that remove the rote and menial tasks from the factory user’s ToDo list. At best remove these types of duties completely from the developers ToDO list. We as solution developers, often have some predictable boring tasks that we have to do over and over again. By exposing automated assets for these types of tasks we save our factory users from completing these mundane and in-variable tasks and let them focus on the variable and more exciting design aspects of the solution they are building. Our factory users will definitely love us for that!

Transfer Domain Knowledge

The design of our automated assets must ensure they capture knowledge and experiences and can be used to make that knowledge available for others. Using the example of a DSLs’ domain model which holds the knowledge about the underlying solution domain and how it can be and should be manipulated. By using the DSL, the user is presented with all the knowledge of how to describe the domain, the parts and relationships of that domain, and the description of the pieces within it.

The important part of this is that the domain model helps the user to figure out the key parts (concepts) in the domain. The domain model (and designer in case of the DSL) guides the user through the process of modelling a solution for the problem by using concepts that make sense to him. In fact, the user learns the domain and the important concepts of it but also how to provide a solution by using these abstractions.

Enforce Compliance and Completeness

When we design the automated assets (and factory as a whole) we encapsulate the architecture of the factory product in the domain model that is used by the factory – this is the model of the product.

We can build automation assets to populate or configure the product model and in this way ensure compliance with the implicit architecture of the product. Because these models have configuration to address their variance, validation can be used explicitly to ensure the factory user has not configured parts of it incorrectly with respect to the whole solution.

Automation assets can also be used to examine the model, check for completeness and use that information to guide the factory user in his next steps he needs to execute in the factory to complete the product.

Full Automation vs. Flexibility

When designing our automated assets we have to keep in mind to keep them limited and focused on one particular concern. It doesn’t make sense to build a huge fully automated asset that covers multiple concerns and generates a lot of unrelated artefacts (i.e. source code and configuration files). It also doesn’t make sense to build a DSL that tries to model to world. If we expose a number of small, highly focussed automation assets we have a much more flexible factory because we can, for example, change the order in which we manipulate the assets, combine them or even decide not to use some automation assets at all to better meet the requirements for the product we have to build.

However, if we go too far with variability and refining our abstractions and automated assets to a more detailed level we might end up with a factory that is less productive because there are more “moving parts” that needs to be configured which also increases the complexity of using the factory.

Finding a balance between full automation and flexibility is absolutely key here!

Usability

Because we are so busy developing our software factory we might forget that building a fully fledged and super cool software factory that generates the perfect product isn’t enough to make everyone want to use it. In the end it is often the factory user that has a huge influence in the adoption of our software factory. This means we have to make it cool to use the thing and make it flexible and intuitive enough to use and improve the experience of building solutions in this domain with it!

You don’t want people to discard you factory because it too hard to use, or discover how to use effectively. Don’t assume they will read the manual!

Promote Adoption

Another thing we have to keep in mind during the development of our factory is the fact that the factory is likely to be used by other less-skilled and less-knowledgeable developers than the ones that building the factory. They might not even have to be developers! This means, we might have an issue here in convincing other development teams to adopt the factory and use that instead of their existing tools, frameworks and utilities they are so familiar with. Or worse, have built themselves!

To get our factory adopted by our audience we have to lower the bar for them to start using the factory. Of course there are multiple ways to achieve this and they will probably differ dependent on the organization structure we are working in. Suppose we are working in an organization where the factory is build by a dedicated team that is known by the several project teams responsible for delivering products. In this example, the factory team can support the project teams in the start up phase of their project to get familiar with the factory and makes sure the project team gets up to speed on using the factory real quick. Once the factory is adopted by the project team responsible for building the product, the factory authoring team can start collecting the feedback from this team to enhance the factory and increase the usability of the factory even better. You see this in the process described in “How would you build it?”

Role Sensitivity

A software factory, just like any other development tool, is potentially used by a lot of project team members (factory users). i.e. Developers, Architects, Testers, User Experience Engineers, Technical Writers etc. To get the full potential from our factory, and factory users, we must make sure the factory targets the audience that is supposed to use the factory in the way they want to use it. In other words before we start developing our factory we need to know what types of users we expect for our factory. Do we only target architects, developers, testers or is it a mix of these types of users that we are aiming at?

Why is it important to know the factory users? Well, each type of user is interested in a different set of concerns and views of those concerns (In factory terminology this is called a ViewPoint). For example, an architect might only be interested in the high level view of the system design while the developer is interested in much more technical details.

Once we know our audience, and those who we are building the factory for, we can attempt to provide them with the right tools, views and level of abstraction to support them do their job in the best possible way and be more productive when using the factory.

Customization

Software development is rapidly changing. New technologies, new paradigms happen almost every day. To be successful in software development we have to keep up with these changes, and therefore our software factory has to too!

In our post “How long will it take?” we have seen that our factory only start to pay back after we have used it for greater than ~5 times, so we better make sure we can customize our factory very easily to meet evolving requirements and additional product variants. This is exactly why we have an iterative development cycle as defined by “How would you build it?”

One of the key requirements of any development tool like a factory is to be flexible enough to address as many use scenarios and development environments as possible. Otherwise its’ use is limited. There are two aspects of customization worth noting here:

Factory Customization

Firstly, there is customization of the factory itself, usually to accommodate a change in architecture or change in variability of the existing architecture that the factory models. This is hard to accommodate if the factory itself presents a closed programming interface. However, this can be accommodated by defining good extensible infrastructures to the architecture so that those using the factory can easily customize it to add or remove features or variability that the factory presents. We see this most when ‘horizontal factories’ are customized to suit certain vertical scenarios, where variability in most cases wants to be predefined, and fixed, and therefore requires removal from the user experience of the factory.

As an example. Consider a (horizontal) factory that builds web services. If I want to ‘verticalise’ this factory and customise it to suit building certain types of financial web services, then it’s likely that for financial web services, I want to pre-define certain Operations Contracts and Data Contracts to use. The Operation and Data Contracts are one point of variability of the horizontal factory, and the horizontal factory provides assets to allow the user to configure them. Now that I want to customise this factory to make a new verticalised version that is financial services based, I want to be able to say that the Operation and Data Contracts are fixed to certain industry standards, and no longer configurable by the factory user.

Product Customization

Secondly, there is customization of the automation assets used to create the product itself. Typically, the scenario here is that you want to alter the automation assets used to transform the model of the product into a physical solution.

As an example. Consider a factory that generates source code as its final product. The source code is determined by the factory in code templates, which are used to transfer the models of the factory into a physical solution. So consider, that the generated code does not agree with you in some way. Often, this needs to be changed to suit an organisations’ coding standards, or include some organisations standard copyright header, or perhaps you want to change the code that is output because you have a more optimal way to write the solution.

Extensibility & Composability

One of the key issues of building factories is in providing the right guidance and solutions to varying requirements. Creating a factory for a large domain increases the chance that the guidance it offers is limited in addressing wider usage scenarios.

For example, consider a factory that builds web services. This factory is likely at some point to define a mapping of the web service’s logical architecture to source code artefacts. In doing that, the factory itself will have to make decisions about what patterns and technologies to use to do that. The assumptions it makes to do that may not be valid for all cases, and it is such cases that may prohibit the use of the factory at all, if the factory user cannot customize or influence how that is done. In smaller domains the chances of meeting this issue are smaller, because the possible solutions are smaller in number. However, even in smaller domains, there can be several solutions that apply to different circumstances. This means that larger domains need to be composed of smaller domains, where each smaller domain could be swapped out for a different implementation.

Composite Factories

In order to practically achieve this we need to be able to compose larger domains from smaller domains, and have the domains interact with each other. Furthermore we need to be able to potentially swap-out each domain with a different implementation.

In factories, this can be achieved by providing extensibility mechanisms that accommodate plugging in and out various other factories to provide the sub-domains. We call these sub-domain factories ‘factorettes’. These extensibility points are key for expanding the use of your factory, so others could provide alternative implementations for the pieces of you factory to suit their needs.

One such use of this is to provide specific technology implementations or strategies for implementation such as the ‘EFx Factory’ does, so that factory users can pick and chose which technology to use to implement the Service Contract or Data Mapping layers, for example. Of course you can choose many other aspects to make extensible in your factory apart from technology. Perhaps you want to allow other factories to provide better views of a part of a domain, or provide better implementations of it as well.

Summary

Previous posts in the Factory 201 series:

• What are they (concretely)? (Edward)
• When would you build one? (Jezz)
• What do I tell my Manager? (Edward)
• How long will it take? (Jezz)
• What would you build? (Jezz)

To continue the thread of actually creating your factory, in this post we will answer the question "With what would you build it?" We will discuss the software factory enabling technologies available today (on the Microsoft platform) and don’t speculate about possible future tooling. In our previous post “What would you build?” we discussed what we would build for our factory, and now we will see how that is mapped and addresses by specific technologies.

Let's have a look at some tooling and technologies and see how we can use that to build our software factories, starting today.   

Automated Actions

Today's generations of software factories, like the ones available from Microsoft p&p, are primarily composed of a related set of automated activities (called recipes) that encapsulate a huge amount of best practices and domain expertise. The recipes often represent repetitive development and design tasks which result in generated source code or configuration files to help accelerate the development of our factory product. Let’s find out what tooling we can use to build these automated activities and how we can expose these, so called recipes, to our factory users.

Guidance Automation Toolkit/Extensions

The Guidance Automation Toolkit (GAT) is a toolkit that simplifies the development of automated actions into re-usable assets and provides mechanisms to make these assets available to architects and developers using Visual Studio. To accomplish this, GAT leverages the Guidance Automation Extensions (GAX) which is actually a runtime environment that expands the capabilities of Visual Studio and makes it easier to integrate and run automate tasks from the Visual Studio environment. So, we are actually using GAT and GAX together to build and run recipes and deploy them as guidance packages. ehhh...? Now what are recipes and guidance packages?

Recipes

Think of a recipe as set of automated instructions, called actions that together represent a specific automated activity or task. These actions are relatively small units of work and can be anything from creating a file, getting the value of a registry key or return the name of the currently selected project within a Visual Studio solution.

A recipe is defined in an XML file in a predefined structure that is interpreted by GAX at runtime. GAX supports invoking wizards for a recipe to gather user input that can be used as variables during the execution of the recipe to create variability in the artefacts the recipe outputs or configures. For example, a wizard can prompt the user to select a file location, ask for a name of a file that gets generated or ask the user to select items from a list. Recipes that don’t require user input can gather the variables from the environment to use in the same way. Recipes also can provide a convenient means of giving the user feedback about the actions being performed.

We can build recipes using GAT, it provides us with a special solution template in Visual Studio to author recipes, it comes with predefined solution structure, example actions and recipes and other utilities that give us a head start in building our own recipes and wizards.

Guidance Package

A Guidance Package is an installable Visual Studio package that contains a collection of recipes (and some their support classes like, converters, value providers, etc.) and is basically our deployment mechanism for the recipes that we built and want to expose in our software factory. In fact, the current generation of software factories as we see today are for the most part simply implemented as pure guidance packages.

That leaves us with another question: "how do we build a guidance package"? Again, this is where GAT comes in. Actually, GAT is a guidance package itself that helps architects and developers build guidance packages. Once we are done developing and testing our guidance package we can create a Windows installer package (MSI) and distribute our guidance package to our users.

Unfortunately, building and testing guidance packages (recipes) isn't as straightforward as we want it to be. For example, building recipes basically comes down to hacking in XML files and isn't supported by any graphical designers. Also the documentation is still limited. This makes the learning curve for developing guidance packages a little steep.

Artefact Templates & Generation

As said, some of the automated activities in our software factory will possibly result in one or more generated artefacts for our solution. These artefacts can be anything in the range from source code, XML files or a piece of documentation. To support this kind of artefact generation in our software factory we can use text templates (also referred to as "T4" templates) and the Text Templating Engine. This technology is included in GAT but also in the DSL Tools which we will discuss a little later in this post.

A text template is basically a text file that contains the (boilerplate) text you want to generate (i.e. C#/VB.NET code, XML, text etc.) that is mixed with control markers (<#, #>) and control blocks. The text between the markers is evaluated and provides the dynamic behaviour to the text template. Control blocks are used to control the processing flow of our templates. In other words they determine what text is output and how the Text Templating Engine navigates over a provided object model. This works very similar to how classic ASP work where you could combine server side directives with the raw HTML code.

Below an example of a T4 template that comes from Service Factory. The text between the markers (in bold) is evaluated and mixed with the static text in the template.

namespace <#= ServiceImplementationNamespace #>

{

   [System.Web.Services.WebService(Namespace = "<#=ServiceContractNamespace#>", Name = "<#=ServiceImplementation#>")]

   [System.Web.Services.WebServiceBindingAttribute(ConformsTo = System.Web.Services.WsiProfiles.BasicProfile1_1, EmitConformanceClaims = true)]

   public class <#= ServiceImplementation #> : <#=ServiceContractInterfaceNamespace#>.<#=ServiceContractInterface#>

   {

   }

}

The most powerful feature of text templates is that they can be invoked from recipes and they can receive variables from our recipes. In this way we can build recipes that collect input from our users or factory environment and use that as input in our text templates to generate artefacts.

A major advantage of using text templates as a means to generate artefacts is that they can be customized very easily, typically to reformat the textual output or enhance it in some way (perhaps to suit an organisations coding standards for example). Text templates are deployed as plain text files so it is straight forward to customize them.

The fact that text templates are so easy to customise makes them perfectly suited for an iterative development approach. We can start with a “hard coded” copy of the desired output and we can identify the variability points in the code. We can then pass some arguments to the template and from that point we can add control blocks to parameterize it and hook into the underlying argument’s object models. One disadvantage of a text template is the lack of designer support like syntax colouring (we added the colouring in the example manually) and intellisense. This makes writing and maintaining large text templates a bit ’un-guided’.

Solution & Project Templates

As we saw in “What would you build?” we can use the solution structure to provide guidance on how to physically relate the artefacts that represent the product of our factory. Unfortunately, defining this (physical) structure is often challenging because of the struggle to define both the deployment model and the logical architectural model of the product in this single structure. Today, the solution structure is all we have (Solution Explorer), so let’s have a look how we can implement predefined solution structures in our factories to describe the physical structure (deployment) of the product.

[The logical architecture is best left to another separate view, which as we will see later we can provide with deep Visual Studio integration]

Solution structures are captured in so called Visual Studio Templates (*.vstemplate files) that are written in XML and are expanded by the Visual Studio template engine. Visual Studio 2005 now supports the creation and packaging of these templates. However, these templates are by default static, and require a separate mechanism to gather variables from the user or environment to enhance them with dynamic values. As it happens, GAT has just that mechanism, and can bind recipes and wizards to these templates.

Let's have a look how we can use this mechanism with GAT to ‘unfold’ certain types of templates:

• Solution templates. These templates unfold an initial Visual Studio solution file (*.sln) and structure that could represent the entire physical structure of the product of our factory.
• Project templates. These templates unfold a project (*.csproj, *.vbproj etc.) within an existing solution.
• Item templates. These templates unfold new project items such as a class files, XML files or any other valid project item.

One of the primary features of GAT is automating the creation of the solution structure. GAT accomplishes this by hooking into the VS template mechanism. This means recipes are allowed to participate in the template unfolding so they can, for example, collect input from the user to customize the names of the template artefacts and take additional actions to further configure the created solution items. For example, write some settings to a app.config file. A perfect example of this is demonstrated in Service Factory where the user is prompted to provide the name of the service, which is then used in the naming of the solution and projects within the solution. We have now discussed GAT and recipes, and learned they are perfectly suited for automating tasks and providing initial solution structures. Let’s look at providing abstractions to our factory and enhancing the design experience and productivity of automating solution assembly. Recipes with wizards often do form a visual abstraction of the underlying part of the product they are automating, but this abstraction is lost when the wizard completes. Let’s have a look at Domain Specific Languages and the DSL Tools and see how they can help enhance that experience and increase the level of abstraction represented in our factory.

Domain Specific Languages

A Domain Specific Language is a custom language that targets a specific domain and helps us describe the problem by using concepts that make sense for the target domain. But how would we build one? This is where the DSL Tools come to rescue. We can use The DSL Tools, (part of VS SDK), to define our domain model and easily represent that domain model with a graphical designer. Together, the domain model and the graphical designer form our DSL that we can integrate into our software factory.  

[You can also provide a textual representation of your language instead of visually, but this is not supported currently out-of-the-box with the DSL Tools yet, unless you are prepared to define that language in XML.]

The DSL Tools also supports artefact generation using T4 templates which means we can write templates that generate (source code) artefacts directly from models we created in our DSL. The DSL Tools is just everything we need to build Domain Specific Languages today.

How would we use the DSL Tools?

So, how would we use the DSL Tools to build a DSL for our factory? Would we build one big DSL and designer that covers the complete product? The answer to this question is probably NO. Although our factory only targets one product (with several variants maybe!) it is likely that the scope of the domain is too broad to fit in just one designer (view). Therefore, we probably want to build a DSL that only targets one concern, component or aspect in our software factory.  For example, we can use a DSL to model a piece of the architecture of the product that is build in our factory or use a DSL to model one of the work products (logical components) in our factory. In this case both DSL's help us presenting the right level of abstraction to our factory user for each of the sub-problems we identified in our factory.

But wait…, we have to make it very clear why we state to keep the scope of the DSL (domain model) limited to only one concern or piece of architecture. In fact, we can think of examples, like the factory schema, that are represented in a ‘larger’ domain model. Mostly, this is because off a limitation in the current technology. The current release of the DSL Tools only supports one view per domain model. In other words, the view that is exposed in our DSL is tightly coupled to the domain model. In future releases of the DSL Tools this limitation will disappear which means we can define multiple views with each view exposing only a part of a, possibly larger, domain model (like the factory schema). Please note that this doesn’t mean we have to build huge domain models in the future, but it will give us more flexibility in our domain model (DSL) design.

[You can find more information on how to use DSL’s in the context of software factories in the article ‘ Bare Naked Languages or What Not To Model ’ from Jack Greenfield ]

Another important limitation of the current DSL tools is that there is no easy way to reference elements in one domain model with those in another, so stitching together domain models to fit larger domains is also challenging. However, there are DSL Power Toys soon to be released called ‘DSL Integration Service’ that enable that cross-referencing.

Another thing to keep in mind is that we don't have to start with a DSL in our factory from the very first beginning. It is absolutely possible that an asset that is implemented as a GAT recipe in a first iteration of our factory evolves in a DSL in one of the next iterations of our factory development cycle.

DSL Advantages

An important advantage of a DSL in our factory is providing a simplified view (abstraction) of the problem domain and to help us to provide architectural guidance in assembling that part. By using specific languages we can present our factory users with the right level of abstraction and use the designer as a way to provide context for our actions and the artefacts we are delivering from our factory. Recipes can also be applied to these designers to manipulate the underlying domain model, and automate the configuration and creating of the elements in these models.

The domain model that is introduced by the DSL (and stored on disk) can be validated and evaluated to determine the progress, validity and completeness of the product we are building. The domain model also provides us with a means to introduce “state” in our factory. We can generate artefacts over and over again for our populated domain model, something we can't do easily with just code generating recipes, where we have to populate the wizards with the necessary information on every run.

A very good example of the usage of a DSL, together with recipes is in the EFx Software Factory, which can be seen here.

DSL Challenges

So, if Domain Specific Languages can add value to our software factories, why haven't we seen them in many factories yet? We think the answer to this question is easy. The reason for that is the lack of "out of the box" integration between GAT and the DSL Tools. Currently, it is hard to use model information in our GAT recipes, without writing the custom code to support this ourselves. Luckily this scenario will be supported soon by means of a DSL Power Toy that becomes available.  

[A more complete description of Domain Specific Languages, the Microsoft DSL Tools, together with an example of an 'real life' DSL can be found in this article ]

Integrated Design Experiences

The technologies that we discussed so far are first class citizens in Microsoft's software factory offering. However, there are more technologies, which are not dedicated “software factory tools”, that can add enormous value to our software factories.

Be warned, these technologies are a little more difficult to use than those described earlier.

Tool-Windows and Custom Editors

One of the things we will notice when we are building software factories is that there will be situations where the decoupled GAT/GAX and DSL Tools features simply don't provide the experience we need. For example, we might want to provide a more integrated user experience for our factory users and decide that we need to provide the user with a larger view of the factory product.

To support this product view we might want to build custom tool-windows or specific editors. An example of this can be seen in the EFx Factory that hosts a custom tool-window that provides a ’Product Explorer’ view for the factory in addition to the default ‘Solution Explorer’ view.

We may want to enhance views with the use of customer editors that provide different abstractions and increase the productivity of modifying the view. An example of that can be seen here where we built a custom tool-window to maintain hierarchical data within a DSL.

It's good to know that we can use the VS SDK for all that. This SDK provides us with utilities and documentation that explains, for example, how to build a tool-window and hook that in Visual Studio. The bad part of this is that creating Visual Studio extensions like this isn't always as straight forward as we like it to be. Besides, building just the tool-window (or editor) isn't enough; we also have to make sure it integrates with GAX or the DSL Tools. Unfortunately this isn't something that is described in great detail anywhere so we are on our own in implementing this. 

If you are interested in this type of integration you better make sure you have the VS SDK documentation ready and find your way to the GAT, DSL Tools and Visual Studio Extensibility forums!

Distributed Application and System Designers 

One means of providing a high level view of your product is to leverage the ‘Distributed Application and Systems Designers’ that come with the ‘Visual Studio Team Edition for Software Architects’. These designers, although currently geared to deployment, provide a useful view of applications which can be composed into larger black box systems. If your factory targets the domain of applications represented on this design surface, this should be a natural choice for leveraging in describing your product, rather than creating DSL’s which duplicate this functionality.

Unfortunately, integrating with these designers today is very challenging, although it is possible to provide custom shapes using the SDM SDK (part of the VS SDK). Integrating recipes and solution templates into these designers is more of a commitment and currently not officially supported since this requires integration with a non-public API. However it is possible to a large degree, as demonstrated by the EFx Factory here. Where you can ‘drill into’ the applications on these diagrams and expose DSL’s containing the inner workings of the applications. You can also ‘Implement’ these shapes by using custom GAT project templates.

In the future, this type of integration will become a lot easier and integration between these designers and GAT and other DSL’s will be a better experience for factory builders. This will provide many factories the capability for top-down solution design.

Summary

In this post we have discussed the relevant technologies that we can use to build automation assets for real life software factories today. We haven't talked about technical details, issues, missing features, etc. which of course are there. The technologies are still in their early days and scenarios for software factory use are still maturing. The learning curve is steep now, and that will definitely become easier in the coming years. However, it is absolutely possible to build some very good factories with the above described technologies. So, if you are interested in factories you better start looking at these technologies right away! 

Now that you know what to do, and what technologies to use for that, the next post in this series “How would you build it?” addresses how to put all these assets together in a process of assembling your factory.

Previous posts in the Factories 201 series:

What are they (concretely)? (Edward)

When would you build one? (Jezz)

To continue the factory 201 series let's discuss a next question: What do I tell my manager? Or maybe, how do we explain the benefits of software factories to the management to justify moving to a software factories approach.

Before we ask the management to move to a software factories approach we have to explain what the value proposition of a software factory is, and what is to be gained from a software factory approach. As we all know, most managers are not interested in technical details so we have to focus on the topics that are important for them! However, the value that we can get from this approach requires a change in the way we develop software today, and this requires a firm commitment from management to achieve success with this approach. Let’s have a look.

Please note that in this post I am using my own experiences from working for a System Integrator. These types of companies are building and using software factories in-house to deliver products for customers. Some of the topics I describe might be (slightly) different for other type of companies, for example for companies that build factories and sell them to their customers without using them in-house.

Value Proposition

The first step might be to tell them what a factory is, we can use the “What are they (concretely)?” post for that. The next step to get them really interested is to explain the benefits of software factories.

Increase Productivity

Probably the easiest benefit to explainis increase of productivity, decrease in cost. Factories automate parts of our development and design work and as a result of that the development team will be more productive during the development phase of the product, and that reduces development costs. But there is more!

Predictable, Consistent Results

Factories will also standardize the product that is developed and therefore we need less time for design and proof of concepts. Another benefit is the improved and consistent quality of the product that is built by the factory. After all, the assets for the factory are built by domain experts and remove thedependency on the very few highly skilled resources (sometimes referred to as ‘local heroes’) or over-resourced projects, to deliver the same high quality work.  A lot of companies are working on improving their software delivery process, reaching a certain CMMI levels and software factories can definitely help with that.

Re-use of: Skills, Resources, Value

Factories use abstractions to simplify the development of solutions. They encapsulate domain specialist experiences and knowledge into the assets behind these abstractions. As a result of that we don’t need to constantly staff each project with very best domain experts, since the assets themselves contain the required experience to solve the problem for us in the best way. Basically, the factory provides a collection of domain skills and experience we can use on each project. These assets represent the actual captured value of our skilled resources.

Furthermore, it’s possible we can re-sell these assets to other companies who may also find getting skilled resources a challenge.

Competitive Advantage

All of the above mentioned benefits are somewhat related to the product that is delivered from the factory, the resources we need for that and are mostly focused on the internal organization. Let’s have a look at some benefits that are recognized externally by investing in factories.

Software factories are hot and will probably become even hotter when they reach the larger community. This is obvious because the value proposition is so strong! Therefore companies can use the presence of software factories in marketing campaigns, and use them as a means to communicate their effectiveness and professionalism to their customers. They can display their innovation in this space and can use that in their recruitment activities to attract senior and skilled professionals.

Resource Attainment

Of course, in the end, our company can also leverage the factory approach to help solving its own resource issues. We can use the best resources to do the most challenging work and actually built the software factories. In this way we continuously challenge these resources, instead of making them participate in ‘boring’ project work and ordinary tools, to make it more likely they will stick with the company. The less experienced resources can become the factory users and be way more productive as they were before.

Commitments

Of course, after telling all of the above described benefits to our management they can't wait to adopt the software factories approach, but wait! There is more we have to tell them to make sure the adoption of software factories within our company has a fair chance to succeed.

Dedicated Resources

We already discussed the requirement for dedicated resources in a previous post (“When would you build one?”) but we cannot stress enough that you have to use the very best experienced, skilled architects and developers to build your factory. They have to be domain experts for the product the factory outputs but also have experience in software (tool) development in general.

However, we all know that the very best resources are the first resources to be pulled off a project and put on another. Since the value of factory assets (and therefore the factory as a whole) directly depend on these resources, it is critical they must not be removed from the factory development project. We all know, and so do the managers, that it’s our customers, market, and organizational culture who decide the allocation of our resources internally. To mitigate this we have to change these conditions to free ourselves from this powerful force to be able to keep these resources focused. The factory development project needs to be separated from the core business development projects, shipping ‘ordinary’ products, and requires a new organizational structure, cost structure, goals, culture and market to succeed. Management needs to support this.

Return on Investment

Return on investment is about profit and growth and thus more important than anything in the world for our management. We have seen the benefits that factories can bring us but before the management is willing to invest in software factories they probably want to know how fast they get their return on investment. After all, why would they want to invest in something that never pays back?

As we have already seen in our "When would you build one?" post, we probably need to build more than ~5 products with a software factory before it pays back so we better make sure we invest in a factory that is likely to be used more than once. We will discuss how and when this investment is returned in one of our future posts “How long will it take?”

Of course, the expectations on the ROI will be different than the established expectations for existing ordinary product development. So, there again, this calls for a separate organizational structure with its own values on what to expect from the product the factory delivers.

Not a Big-Bang Effort!

We have to make sure we keep the initial investment costs as low as possible. Therefore, trying to build a huge, fully featured factory from scratch in one attempt doesn't make sense, not only from a technical perspective but also from an investment perspective. We need to see a return on each iteration of the factory development.

To be successful, a progressive, iterative approach is preferred. The best thing you can do here is to start with small increments in automation of the product your factory creates. With each increment we identify the variable parts of the product and design those into the next iteration. This way, management can see an incremental return on every cycle.

To ensure your factory is applicable to more scenarios in its market, make sure management promote its use in many projects, even if the factory covers only a (relative) small part of the solution that is built. This way your factory can quickly identify many valuable scenarios of use, and support those in its evolving design.

Another option we have to minimize the initial investment costs is to re-use and customize an existing factory. For example, Microsoft patterns & practices released some (early examples of) software factories that are totally free and ready for use. It only takes us a few days to get familiar with these factories. After that we are ready to start using them in real life projects. Our project will benefit from the factory that we are using and in the meantime we will build experience with using software factories in our software development process. All of this with minimal investments but the experience we will gain from that will be invaluable for our future factory development activities. 

Measuring Success

One of the first questions we will have to answer when asking for an investment in factories is: "how do we measure the success of the factory?” In other words, we know the factory is supposed to increase our productivity, improve the quality and degree of standardization, but how do we notice and when will we see our first results? We have to make sure the management knows what to expect and therefore we have to specify this upfront.

For example, we can state the factory delivers a 5 % productivity increase during the development phase for projects larger than 2000 man hours, we can maybe save 80 hours during the design phase because we can use a standardized architecture documents and need less time for building a proof of concept. Further we have to specify when we can expect these results. For example, do we benefit from the factory in our first project or is it more likely this will happen in the second or even third project because the factory needs some improvements, people have to get familiar with it, etc. Of course, all of this depends on the state of the factory itself, the project, etc. but it is crucial that we use this kind of data to validate the effectiveness of the factory and the investment we are making in software factories.

Promote Use and Feedback

Factory development isn't something we do as a "stand alone" activity. After all, the factory is supposed to be re-used by a lot of projects and different project teams within our company. To make sure we don't suffer from the "not invented here" syndrome we have to make sure the factory authoring team, but also the management communicates openly about these initiatives. We have to tell our colleagues about our initiative, demo an early drop or even better, let them play with it and ask them to provide feedback, tips hints, etc.

Support Factory Adoption

Another thing we can do to increase the adoption of our factory is to help people use the factory on their project. Tell them a member of the factory authoring team is available at the start of their project to support them in using the factory. This will give them a head start, not only from the productivity and guidance they get from the factory itself but also from a domain expert that demonstrates them how to best use the factory in their situation. From experience we can tell this is something that is really appreciated by most projects. We can also ask projects that we supported to provide feedback to the management and tell them how the factory helped them in their project. This will definitely help us in marketing our factory approach within the company.

Summary

In my previous post "Software Factories: Where to start" I tried to list some resources that might be interesting for new comers in the software factory space. I have been thinking about factories and also participating in this space for some time now and I recently realized that, although there is a lot of buzz around software factories it is still very difficult to enter the world of software factories.

It might sound simple; install GAT and the DSL Tools and build your first factory but unfortunately, from what I experienced, it isn't that easy (it might be me of course!). I had (and still have) a lot of questions like for example: "What is a software factory", "When would you build one", "How would you build one", "What tools would you use" and many, many more.

It's therefore that  I decided to share some of my thinking's about factories to hopefully provide some answers to people that are about to join the software factories space. I am not a very formal guy so will try to write about these topics from a (factory) developer or (factory) architect perspective and keep it practical and understandable.

To be honest I am not sure I have the answers for all questions just yet but we will see where this leads too. Regular readers of this blog might have noticed that I discuss a lot of my software factory related work with Jezz. He did a lot of interesting posts on factories lately so hopefully he reads this post and will comment, or even better, add some content when he sees me struggling in my attempts (don't feel forced Jezz, but know that I am counting on you to help me out! :-)).

Ok, now we know the plan, let's make sure the title of this post makes sense and start with the first question:

What are they (concretely)?

If you were looking for a formal definition of what a software factory is, you would find one in the Software Factories” book written by Jack Greenfield and Keith Short. In this book, a software factory is defined as:

“A Software Factory is a software product line that configures extensible tools, processes and content using a software factory template based on a software factory schema to automate the development and maintenance of variants of an archetypical product by adapting, assembling and configuring framework-based components .”

Fair enough, but do we, as a developer, know what to build by just reading this definition? Not me. The patterns & practices team came up with a more pragmatic definition, that is in fact more accurate for the factories being built today:

“A software factory is a structured collection of related software assets. When a software factory is installed in a development environment, it helps architects and developers predictably and efficiently create high-quality instances of specific types of applications.”

But even this definition, does not really tell you what the factory itself is in concrete terms, at least not in enough detail to help you get started in building one. So we are going to use a much more simplified, and concrete definition that is more to the point from a developers perspective, describing what it actually is and how it physically works. Then from this, you will get a basic understanding of how the definitions above describe more precisely the moving parts of the factory.

[Please note, that this definition is being used here for the purposes of helping you understand the context of this series on building factories, and is not a formal and general definition of what a software factory is when it is finally shipped. The following definition only focuses on the concrete automation part of a factory. Inside the box of a real factory you will find other physical guidance assets like documentation, samples, reference implementations, training media etc. and together these make up a complete software factory.]

It’s a Tool!

First of all, a software factory is a software development tool. This tool is used to automate the assembly and/or configuration of a software solution that is addressing a well known and described, specific problem domain. It’s here where we see our first difference between a software factory and another development tool like for example Visual Studio.NET. A software factory is targeted at only a small problem domain while we can use Visual Studio.NET to basically build anything we want (of course within limits).

Configures and runs within Visual Studio

Although there is a difference between software factories and Visual Studio.NET (for the Microsoft platform) in the size of the problem domain they target, they are also very much related. This is because a software factory runs within Visual Studio.NET and basically configures this environment to target the development of a specific problem domain. Things like menus, windows, editors, layout etc.

For example, think of the Microsoft p&p Service Factory that configures Visual Studio to develop web services. This (early generation) software factory adds some menus, wizards and templates to the standard Visual Studio environment (Solution Explorer) to help developers build services.

Uses Abstraction

Another characteristic of a software factory is that it provides abstractions to simplify the problem domain it targets. An abstraction is something that simplifies the thing it describes and only focuses upon on only the important characteristics or variable parts of it.

In factories you can provide abstractions using several techniques like frameworks, specialized editors, wizards, models and diagrams.

Although providing abstractions is one of the most important requirements of a software factory, we haven’t seen a ‘publically available’ factory that uses all of these techniques - but I am sure we will see more in the future.

If you are interested in examples of abstractions that are used in factories, you might want to read this MSDN article about the EFx factory. The DSL that is used in this factory is a perfect example of the usage of abstraction.

Creates a Product

Another very important aspect of factories is its output. A software factory outputs a generated solution for the problem domain it targets. The product of a factory is what it outputs. A software factory doesn’t necessarily output source code. For example, it’s also possible that a factory generates some configuration files that are used to configure an existing framework, some documents, or whatever makes sense to the problem domain it targets. They can also just configure other products which solve a particular domain. One thing to keep in mind is, that is likely that the factory output needs some manual work or customization before it is ready to use.

Customization

There is also another type of customization that is very applicable to software factories. This is, the customization of the software factory itself instead of the output it generates. This type of customization leads to an iterative approach where we start with a small factory that iteratively increases. It is this type of customization that we use to address a new concern in the factory or, solve a defect in the factory product, or customize our assets to suit your development requirements and corporate standards.

Summary

So, here we are. I tried to explain what a software factory really is by describing some off its key characteristics without using common software factory terms like “product line”, “schema”, “variants”, etc. Because of that I might be missing some important stuff but I am sure I can make up for that in a way that make sense in future posts when I cover some other, software factory related topics.

For those of you who can’t wait for that, and like to read a little bit more about a software factories definition in more software factory related jargon, I can recommend this post from Jezz.

If there are any software factory related questions that you might have and want me to answer please let me know by commenting on this post. I cannot guarantee I have the answers but we can at least discuss about it and try to find the answer together.

To be continued…

As it looks like now, 2007 will be the year where software factories get a lot of exposure and attention. We have seen quite a lot of activity in this space in the second half of 2006 but there is definitely more to come! This also means that there will be a lot of new people entering the software factory space in the next coming months. For those of you who are not that experienced yet with software factories I decided to create a short list with some interesting links to existing software factories, forums, blogs, etc. Hopefully, this overview gives you a head start in entering the exiting world of software factories.  

Available software factories

If you are a practical guy (or girl) and like to get your hands dirty as soon as possible you might want to get yourself an existing software factory and start experimenting with it right away. In that case, check out the following offerings from Microsoft P&P. Clicking the links below will bring you to the MSDN landing page for each of the factories were you will find detailed information, links to the community and links to the download pages.

1. Web Service Software Factory
2. Web Client Software Factory
3. Smart Client Software Factory
4. Mobile Client Software Factory

You may also want to subscribe to the blogs of some of the P&P Product Managers: Don Smith, Tom Hollander and Eugenio Pace to keep you up to date with the latest news and future directives of the P&P factories. If you are interested in an indication of the "release dates" of the (future) P&P deliverables, check out their brand new "Upcoming Releases" page. 

Software Factories Tools

After experimenting with these existing factories for some time you might want to modify them (to make them better suite your requirements), extend them with some Domain Specific Languages or completely build your own factories from scratch. In that case you have to check out the technologies that drive the current wave of software factories.

First of all, we have the Guidance Automation Toolkit which you can use to create “packages” that can be installed in Visual Studio.NET and basically represents software factories as we see them today. You can download it from here and you can ask your questions about it (which you will definitely have ;)) here.

Second, we have the DSL Tools that you can use to create Domain Specific Languages that you can include in your software factories. You can download the DSL Tools from here and ask your questions about it here.

If you are interested in the combination of the Guidance Automation Toolkit and the DSL Tools you might want to read this article that describes some of the possible integration scenarios. (the library that is used in this article isn’t available yet, but will be soon!).

If you plan to start building your own factories, based on the above mentioned technologies, you also might be interested in the Clarius Software Factories Toolkit . This toolkit adds some extra features to the existing technologies to make it easier to develop your own factories. 

Background information, theory and concepts

Of course, there are also people who are really interested in the theory and the concepts behind the software factories strategy. If that’s you, go get yourself the Software Factories book written by Jack Greenfield and Keith Short . Once you have read this book you realize that we are still at the very early beginning with software factories and there is a lot of improvement to be made in the coming years. Some time ago I have written a post myself about some of the limitations in the first wave of software factories. As you can read there today’s software factories lack a model, schema and a lots of other stuff.

If you are interested in the jargon that is often used in the software factories space, check out this “Software Factories ABC” that I co-authored with Jezz . You better subscribe to his blog if you are interested in software factories because Jezz is an active factory blogger and has some really interesting posts about models, schema and tools. If you are still hungry for more information you can also check the software factories landing page on MSDN for more information.

So, we are done with the list. Of course there is a lot more interesting information about software factories but I am sure you will find them when you start your journey by experimenting with the currently available factories and reading the information I linked too in this post. Hopefully this list can be off any help to "software factories newbies" to get familiar with this very interesting technology as soon as possible.

Happy factories!

Update: also check out the Software Factories Swicki as the online search enigne for all factory related stuff!

The article about Domain Specific Languages I wrote together with Christian Weyer for CoDe magazine is now online. It’s called “Introducing Domain Specific Languages” and briefly describes the Software Factories initiatives before it focuses on Domain Specific Languages in general and our DSL for the remainder of the article.

At this moment only a piece of the full article is available for non-subscribers. This means you either have to subscribe to CoDe magazine or save the link and get back in 2 months when the full article is available for all site visitors of the CoDe magazine.

 Hope you like it!

My friend Jezz started a new series about the work he is doing on building a Toolkit for developing enterprise distributed applications and services. I know some off the details of his work and this is quite interesting. Jezz is just like my pasionate abouth DSL, GAT, and all related stuff. Let’s see what he has to say in his coming posts...