In my previous post "Software Factories: Where to start" I tried to list some resources that might be interesting for new comers in the software factory space. I have been thinking about factories and also participating in this space for some time now and I recently realized that, although there is a lot of buzz around software factories it is still very difficult to enter the world of software factories.

It might sound simple; install GAT and the DSL Tools and build your first factory but unfortunately, from what I experienced, it isn't that easy (it might be me of course!). I had (and still have) a lot of questions like for example: "What is a software factory", "When would you build one", "How would you build one", "What tools would you use" and many, many more.

It's therefore that  I decided to share some of my thinking's about factories to hopefully provide some answers to people that are about to join the software factories space. I am not a very formal guy so will try to write about these topics from a (factory) developer or (factory) architect perspective and keep it practical and understandable.

To be honest I am not sure I have the answers for all questions just yet but we will see where this leads too. Regular readers of this blog might have noticed that I discuss a lot of my software factory related work with Jezz. He did a lot of interesting posts on factories lately so hopefully he reads this post and will comment, or even better, add some content when he sees me struggling in my attempts (don't feel forced Jezz, but know that I am counting on you to help me out! ).

Ok, now we know the plan, let's make sure the title of this post makes sense and start with the first question:

What are they (concretely)?

If you were looking for a formal definition of what a software factory is, you would find one in the Software Factories” book written by Jack Greenfield and Keith Short. In this book, a software factory is defined as:

“A Software Factory is a software product line that configures extensible tools, processes and content using a software factory template based on a software factory schema to automate the development and maintenance of variants of an archetypical product by adapting, assembling and configuring framework-based components.”

Fair enough, but do we, as a developer, know what to build by just reading this definition? Not me. The patterns & practices team came up with a more pragmatic definition, that is in fact more accurate for the factories being built today:

“A software factory is a structured collection of related software assets. When a software factory is installed in a development environment, it helps architects and developers predictably and efficiently create high-quality instances of specific types of applications.”

But even this definition, does not really tell you what the factory itself is in concrete terms, at least not in enough detail to help you get started in building one. So we are going to use a much more simplified, and concrete definition that is more to the point from a developers perspective, describing what it actually is and how it physically works. Then from this, you will get a basic understanding of how the definitions above describe more precisely the moving parts of the factory.

[Please note, that this definition is being used here for the purposes of helping you understand the context of this series on building factories, and is not a formal and general definition of what a software factory is when it is finally shipped. The following definition only focuses on the concrete automation part of a factory. Inside the box of a real factory you will find other physical guidance assets like documentation, samples, reference implementations, training media etc. and together these make up a complete software factory.]

It’s a Tool!

First of all, a software factory is a software development tool. This tool is used to automate the assembly and/or configuration of a software solution that is addressing a well known and described, specific problem domain. It’s here where we see our first difference between a software factory and another development tool like for example Visual Studio.NET. A software factory is targeted at only a small problem domain while we can use Visual Studio.NET to basically build anything we want (of course within limits).

Configures and runs within Visual Studio

Although there is a difference between software factories and Visual Studio.NET (for the Microsoft platform) in the size of the problem domain they target, they are also very much related. This is because a software factory runs within Visual Studio.NET and basically configures this environment to target the development of a specific problem domain. Things like menus, windows, editors, layout etc.

For example, think of the Microsoft p&p Service Factory that configures Visual Studio to develop web services. This (early generation) software factory adds some menus, wizards and templates to the standard Visual Studio environment (Solution Explorer) to help developers build services.

Uses Abstraction

Another characteristic of a software factory is that it provides abstractions to simplify the problem domain it targets. An abstraction is something that simplifies the thing it describes and only focuses upon on only the important characteristics or variable parts of it.

In factories you can provide abstractions using several techniques like frameworks, specialized editors, wizards, models and diagrams.

Although providing abstractions is one of the most important requirements of a software factory, we haven’t seen a ‘publically available’ factory that uses all of these techniques - but I am sure we will see more in the future.

If you are interested in examples of abstractions that are used in factories, you might want to read this MSDN article about the EFx factory. The DSL that is used in this factory is a perfect example of the usage of abstraction.

Creates a Product

Another very important aspect of factories is its output. A software factory outputs a generated solution for the problem domain it targets. The product of a factory is what it outputs. A software factory doesn’t necessarily output source code. For example, it’s also possible that a factory generates some configuration files that are used to configure an existing framework, some documents, or whatever makes sense to the problem domain it targets. They can also just configure other products which solve a particular domain. One thing to keep in mind is, that is likely that the factory output needs some manual work or customization before it is ready to use.

Customization

There is also another type of customization that is very applicable to software factories. This is, the customization of the software factory itself instead of the output it generates. This type of customization leads to an iterative approach where we start with a small factory that iteratively increases. It is this type of customization that we use to address a new concern in the factory or, solve a defect in the factory product, or customize our assets to suit your development requirements and corporate standards.

Summary

So, here we are. I tried to explain what a software factory really is by describing some off its key characteristics without using common software factory terms like “product line”, “schema”, “variants”, etc. Because of that I might be missing some important stuff but I am sure I can make up for that in a way that make sense in future posts when I cover some other, software factory related topics.

For those of you who can’t wait for that, and like to read a little bit more about a software factories definition in more software factory related jargon, I can recommend this post from Jezz.

If there are any software factory related questions that you might have and want me to answer please let me know by commenting on this post. I cannot guarantee I have the answers but we can at least discuss about it and try to find the answer together.

To be continued…

As it looks like now, 2007 will be the year where software factories get a lot of exposure and attention. We have seen quite a lot of activity in this space in the second half of 2006 but there is definitely more to come! This also means that there will be a lot of new people entering the software factory space in the next coming months. For those of you who are not that experienced yet with software factories I decided to create a short list with some interesting links to existing software factories, forums, blogs, etc. Hopefully, this overview gives you a head start in entering the exiting world of software factories.  

Available software factories

If you are a practical guy (or girl) and like to get your hands dirty as soon as possible you might want to get yourself an existing software factory and start experimenting with it right away. In that case, check out the following offerings from Microsoft P&P. Clicking the links below will bring you to the MSDN landing page for each of the factories were you will find detailed information, links to the community and links to the download pages.

1. Web Service Software Factory
2. Web Client Software Factory
3. Smart Client Software Factory
4. Mobile Client Software Factory

You may also want to subscribe to the blogs of some of the P&P Product Managers: Don Smith, Tom Hollander and Eugenio Pace to keep you up to date with the latest news and future directives of the P&P factories. If you are interested in an indication of the "release dates" of the (future) P&P deliverables, check out their brand new "Upcoming Releases" page. 

Software Factories Tools

After experimenting with these existing factories for some time you might want to modify them (to make them better suite your requirements), extend them with some Domain Specific Languages or completely build your own factories from scratch. In that case you have to check out the technologies that drive the current wave of software factories.

First of all, we have the Guidance Automation Toolkit which you can use to create “packages” that can be installed in Visual Studio.NET and basically represents software factories as we see them today. You can download it from here and you can ask your questions about it (which you will definitely have ;)) here.

Second, we have the DSL Tools that you can use to create Domain Specific Languages that you can include in your software factories. You can download the DSL Tools from here and ask your questions about it here.

If you are interested in the combination of the Guidance Automation Toolkit and the DSL Tools you might want to read this article that describes some of the possible integration scenarios. (the library that is used in this article isn’t available yet, but will be soon!).

If you plan to start building your own factories, based on the above mentioned technologies, you also might be interested in the Clarius Software Factories Toolkit. This toolkit adds some extra features to the existing technologies to make it easier to develop your own factories. 

Background information, theory and concepts

Of course, there are also people who are really interested in the theory and the concepts behind the software factories strategy. If that’s you, go get yourself the Software Factories book written by Jack Greenfield and Keith Short. Once you have read this book you realize that we are still at the very early beginning with software factories and there is a lot of improvement to be made in the coming years. Some time ago I have written a post myself about some of the limitations in the first wave of software factories. As you can read there today’s software factories lack a model, schema and a lots of other stuff.

If you are interested in the jargon that is often used in the software factories space, check out this “Software Factories ABC” that I co-authored with Jezz. You better subscribe to his blog if you are interested in software factories because Jezz is an active factory blogger and has some really interesting posts about models, schema and tools. If you are still hungry for more information you can also check the software factories landing page on MSDN for more information.

So, we are done with the list. Of course there is a lot more interesting information about software factories but I am sure you will find them when you start your journey by experimenting with the currently available factories and reading the information I linked too in this post. Hopefully this list can be off any help to "software factories newbies" to get familiar with this very interesting technology as soon as possible.

Happy factories!

Update: also check out the Software Factories Swicki as the online search enigne for all factory related stuff!

Service Factory V2, is there!  This release provides some great guidance for building service for WCF and ASP.NET (ASMX). Get it here and make sure to post any feedback and/or questions on the Service Factory community site!

Ok, I have been tagged by Erwyn so here is my list of 5 things you might not know about me.

1. When I got my first job back in 1995 I started programming in Microsoft Visual Foxpro 3.0. At that time I found it SO much better than all the other languages (dBbase III+, Pascal and even Cobol) I programmed in before and was really happy with this decision of my employer to use this. Visual Foxpro was a data centric, object-oriented environment and I continued programming in ( newer versions of) Visual Foxpro till the moment the first CTP of .NET came out. From that moment I started programming C# and never went back. When I had my first look at LINQ I was very surprised to see the “Visual Foxpro data centric approach” being integrated in .NET.
2. I am a KOI keeper! I spend a lot of my spare time on taking care of my 10 beautiful KOI. Because KOI require an excellent water quality I am using several filtration units, millions of Nitrosomonas, UV light and air pumps to keep my pond clear. (some examples of these beauties, unfortunately not mine)
3. I am married for almost 5 years now with my wife Miranda. I live in the Netherlands in a small town named Hoorn and together with my wife I have a beautiful daughter, named Emily, of almost 2 years old. 
4. In the evenings, when I am not busy with my KOI or family, I spend most of my time with my laptop on the    couch. I do not watch TV a lot (only some movies) and don't like to sit behind a desk when I am not at work.  So, all my blog reading, sending mail, MSN, etc. is done from my very comfortable couch.
5. At the age of 17 I started with fitness. Unfortunately I got really addicted to this and fitness slowly became bodybuilding! In that period I was in the gym 7 days a week, ate 3 kilos of cooked potatoes, 300 gram steamed vegetables, a grilled chicken breast fillet, 2 pieces of fruit, some protein shakes a day. At the top of my "bodybuilding carrier" I weight 84 kilogram (184.8 lbs) which is pretty my for a small guy (approximate 5 feet 6 inches) like me. After some years, when I got my first job at a large Dutch Bank, I was aked to wear suits at work. So, I went to the store to buy some new suits. It was at that moment, when I noticed I couldn't buy "normal suits" anymore, I finally realized my body wasn't in normal proportions any more. From that moment I started thinking about quitting fitness (bodybuilding) and luckily after some time I did. (I decided not to post any pictures to make this story not even more embarrassing )

I tag Jezz, Rene, Dennis, Carlo, Christian, Mark

Some time ago I started building a new Domain Specific Language for modeling message and data contracts. Building the DSL was pretty straight forward so it didn't take me too much time to build the beauty. However, during the debugging and testing phase of that DSL I noticed that the user experience of the DSL wasn't too good. For example, this particular DSL lets you model a data contract. Each data contract has a relation with one or more data members (implemented as compartment items) and for each data member you have to set its name, type, order and required properties. So, to add a data member to a data contract in my DSL I first had to do two right mouse clicks, after that I had to type the name of the data member, after that I had to switch to the properties windows and type the various values of the data members properties. This just didn't feel too good, especially when modeling a number of message and data contracts in a row. At that time I decided not to use the DSL, simply because of the, in my opinion, poor user experience.

Just recently, during some other Domain Specific Language related work I noticed again that I had some trouble getting the user experience right for modeling "relations" between model elements. Again, it was the same number of mouse clicks that bothered me (maybe it's just me?!). So, it was about time to try and find a solution for this "problem". I soon decided to build my own custom tool window specifically for my DSL but didn't know yet how to represent the "relational data" in that tool window. Luckily, in one of my talks with Jezz he came up with this brilliant idea to use the "VirtualTreeControl" (Microsoft.VisualStudio.VirtualTreeGrid.dll) for this purpose. He already used this control in the EFx Factory as you can see here. I have to be honest, this control is a real bitch to figure out, it isn't documented and I don't even know if it is officially supported but with some help of Jezz and a wrapper he built for it specifically for DSL relations, I got this working for my message-/data contracts DSL.

With this control you can show hierarchical domain relationships, and each row can have child rows, and you can display whatever columns of data you want from the relationships and domain classes.

As you can see in the picture below I can now maintain my "relational data", for my data contracts (in this case data members) in my custom tool window ("Data Contract Editor". I can easily add new data members and set their properties without having to switch between the mouse and keyboard over and over again. I think this control really rocks and I absolutely love its user experience.

The control is especially useful for exposing domain relations and domain classes which don’t have shapes or connectors on the diagram. For example, children of domain classes displayed in compartments. Therefore the control reduces shape ‘clutter’ on the diagram.

I like to consider tool windows and/or editing controls like this as "out of the box" features for future releases of the DSL Tools! In my opinion, it will dramatically improve the usage scenarios of DSL build with the DSL Tools.

For my current assignment, which is related to integrating existing and developing new Software Factories in our software development process, I decided to have a look at project GlidePath. According to the GlidePath website this project will deliver guidance on how to build software for Vista.  It delivers a web based repository of software factories that can loaded into the Visual Studio environment.  One of the very interesting features of GlidePath is the possibility to update the Software Factories over an RSS feed. It’s actually this feature that made me investigate GlidePath to see if we can use this or a similar approach in our own development process.

Once installed, you can use the Glidepath guidance package to create a Visual Studio solution structure. Instead of generating a full blown solution structure, like for example Service Factory does, Glidepath provides you with a rather empty solution.  At that moment the solution doesn’t reflect the actual product at all but rather provides you with the possibility to decide what factory to use in this solution structure (and thus what product to build?).  In this release of Glidepath you can only select one factory (MicrosISV factory) but I assume this will change in the future. Once I had added the MicroISV factory to my Glidepath solution I was surprised to see a lot of viewpoints popup in my solution structure (at this moment represented as .txt files).  From what I understood from Glidepath the idea behind these viewpoints is that you can select and configure them to specify how to build your product in the factory.  Glidepath will create new projects or enable packages (factories) in your current solution based on the viewpoints you configure for your product. Currently, Glidepath populates the MicroISV factory with viewpoints like “DataAccess”, “Deployment “, “Blogging”, “Legal”, etc.

Figure 1. Glidepath solution structure

After playing with Glidepath for a while I suddenly realized that I didn’t had a clue what kind of product I was actually building in my factory. I could see that my solution was populated with a project for “blogging” and project for “CSharp-WinForms-UX” because I selected these viewpoints for this solution.  However, from looking at the solution structure I wasn’t able to tell whether I was building a “blogging service”, a “blogging site” a “windows client for my blogging service” or whatever. Of course I understand this is related to this very early release of Glidepath and therefore lack of implemented packages, but I realized this was caused by the fact that the Glidepath factory doesn’t  have any notice (yet) about the product itself and therefore couldn’t tell me either what product I was building.

I was interested by this and tried to understand what this would mean for me if I had to implement (build a guidance package) one of the viewpoints for MicroISV. How would I know about the underlying product my viewpoint will be used on? Is it even necessary to have knowledge about that or can we generalize viewpoint implementations so they can be used on all products we are building? To be honest, I don’t have the answers for that at this moment.  My current thinking is that it would be helpful to have a formal description of the product I am implementing my viewpoint for but as said, I don’t have the answers yet.

A related question that interests me is: “what comes first?” Let’s say I was asked to build a factory for product X. Would I start with defining the viewpoints for my factory (security, deployment, etc.) or would I start by describing the basics of product X first? Or would I do this simultaneously?

Logically I would say, I first need a full picture of the product (product X) I am building the factory for.  Ideally, I would describe this product at a conceptual level (product model) in a way that makes this knowledge easily accessible and understandable for other people.  Once I know what I am talking about, and thus have understanding of the problem domain, I can decide what viewpoints are important, how to provide automated guidance for that, etc.

If we have captured the knowledge about the product we cannot only use this to author the factory but also to guide the factory user through the product at runtime. The factory can use the product model to communicate with the factory user in terms that make sense to the factory user and are related to the problem domain he is developing software for.

Currently Jezz and I are working on an experimental software factory authoring approach using a tool called the Factory Product Model (FPM).  I will not explain this approach or tool in detail in this post but if you are interested in more details you can read this, this, this and this post.

In this experiment, we are trying to validate whether a formal description (model) of the product the factory is building is a valid starting point for defining the factory and what it builds.

As far as I understand now, viewpoints will definitely play an important part in factory authoring and usage but what I am trying to find out is what impact they have and how they possibly relate to something like FPM.

I think specifying viewpoints is all about (some of) the requirements of the product we are building in the factory and therefore impact the type of guidance that is needed in the factory.

After all a viewpoint is what you get when you associate a work product, to an assets and a set of activities. Therefore viewpoints and work products are tightly related. It is just a question of which one do you define first?

I would be very interested in any ideas about this topic. What do you think? Do we need a product first approach (an FPM)? How does this relate to the Glidepath viewpoints approach I described? Do we need both? Do we need them in a specific order?

Let me know!

Two days ago, my colleague Rene and I gave a presentation on an internal (Dutch) LogicaCMG event for developers about Software Factories. In this session we first covered some general Software Factory stuff and after that focused on Service Factory to demonstrate the power of this, what I call, first wave of software factories from Microsoft P&P.

In our timeslot we had to compete with three other sessions but we managed to get 160 developers/architects to attend our session. It turned out that, at least on paper; we had the most popular session! I am still not sure about the reason for that. Hopefully this is because the Software Factory topic is hot. Another possible reason for this is that we might have fooled people with the title of our presentation. We choose “Software Factories, how to get a SOA in one hour” as the title for our presentation. In Dutch SOA means “Sexueel Overdraagbare Aandoening” which corresponds to “Sexual Transmitted Disease” in English. (However, I can’t believe that people really think you need an hour to get yourself a “SOA”.)

Besides a little typo in the demo (which of course caused a compile error) the presentation went quite well. Luckily, we had a complete (working) solution prepared so we didn’t have to spend anytime during the presentation on trying to find the typo.

During our presentation we demonstrated how to write a service from scratch (within one hour) by using Service Factory. In our demo we also showed how to use this domain specific language integrated with Service Factory to overcome some of the “limitations” of the wizard based approach in Service Factory.

Based on the feedback I got after the presentation it turns out that people are really enthusiastic about Software Factories, Service Factory and the accompanying Microsoft tooling (GAT and DSL Tools). Hopefully, we managed to get people interested enough in this technology to make them download Service Factory today!

Below some pictures. As you can see we were wearing our cool "P&P shirts Don gave us at the last Service Factory workshop  

In a previous post I explained that Jezz and I are working together to prototype a different “approach” for authoring software factories.

In this experiment we are trying to validate this new approach based upon design ideas refined in the implementation of the EFX factory, and verifying that against the publicly available release of p&p Service Factory.

Basically, we think that we need to be able to let the factory author describe the ‘Product’ that is built by their factory at a higher abstraction level than we do now. We suspect that this approach will greatly help in improving the factory authoring experience and the factory user experience, which therefore simplifies software development as a whole.

Part of this experimental approach is describing the important concepts of the factory product in a so called “Factory Product Model (FPM)”. Below you can see a first attempt to model the Service Factory ‘Product’ in an experimental version of the FPM designer.

To make the example a little more realistic we introduced a few new concepts and made some assumptions.

For example we assumed it is possible to develop different type of services (entity services and process services) in Service Factory (there could well be others too). We did this to illustrate the concept of ‘Product Variants’. Which are basically one type of product the factory creates. Since a factory can create several variants of the product in its product line.

[Think of a ‘Product Variant’ as a very similar product but different in architectural definition, rather than something different in instance values of the concepts.]

Another assumption we made was that the difference between these services only reflects in the “service implementation” part. This is probably not completely true but we are only using these assumptions to simulate the situation where the factory enables us to build different family members of the service product line.

Figure 1. Factory Product Model for Service Factory 

The picture of the FPM for Service Factory shows us that we identified a “Product Variant” for each service type that is supported in (our imaginary) Service Factory. Also we identified the “System” product variant that makes it possible to group together any number of these services. We could have gone with one product variant for all types of services, but we wanted to define some that are different to validate the approach.

You can imagine then, that each product variant would be represented to the factory user as a separate solution template offered to them when they create a new solution using this factory. (In this case we would have 3 solution templates: ‘Entity Service’, ‘Process Service’ and ‘System’.

Looking at the concepts in the model, one interesting thing is the “Service Contract” concept; the (green) concept we are using, reflects that we will be using a custom Domain Specific Language for modelling Service Contracts in our factory. This custom DSL for modelling service contracts is responsible for modelling all the concepts that are defined within the “Service Contract” shape in the Service Factory FPM. This kind of model is called a Work Product Model (WPM), since it defines work products of the FPM. The child concepts of the ‘Service Contract’ concept will be refined in this WPM, and are represented here for logical structure.

The other concepts in the Service Factory FPM reflect common terms used in the Service domain that are therefore part of Service Factory. The concepts in this FPM will be represented to the factory user as a mechanism to “drive’” the factory. This view provides the user with a higher level abstraction than the current “Solution Explorer” view that is used in today’s factories. So instead of being confronted with all the technical details he can develop his services by using concepts that make sense to the service domain.

Figure 2. Example of the Service Factory Product Explorer

To make all of this happen, the next step would be mapping each concept in the FPM to physical solution structure, projects, and project items of Service Factory. Once that is complete, we define the assets (recipes, templates, designers, etc.) that will be used to create and configure these physical artefacts. Once configured correctly, the FPM is then used as a starting point for populating the factory schema for this factory. It already contains the definitions of all the work products, their assets.

Another idea we would like to validate in this experiment is the speculation that we can use the concepts modelled in the FPM to use as reusable parts in other factories. It would be great if we could find a way to just “query” this FPM (and/or factory schema) of a factory, and select the available concepts that we want to use to compose other factories from. You could imagine in this case that the ‘ServiceContract’ concept is self-contained enough to be reused in another factory (complete with DSL model and definitions).

Interested in Software Factories and trying to understand the common terms uses in this pace? Have a look at this Software Factory ABC at Jezz's blog. Please leave a comment if you have anything to add/change and we will be happy to update the list!