What I'm looking for...

In a number of the posts I've been writing on the SOA/BPM conference I've referred to the applicability (or lack thereof) of a given approach to "the problem set" that I'm currently working on. I thought it might be good to describe what it is that I'm looking for and a little bit as to why.

I'm working at an organization with roughly 4,000 employees that is in the process of "drinking the MSFT Koolaid". We are deploying nearly every MSFT product and working hard to bring consistency to the platform both from a services aspect as well as the development paradigm. We are focusing on SAP backend, SQL data store, and SharePoint/Office Suite front-end.

We are also focusing on bringing a consistent story to the workflow problem, and, by "workflow" I really mean (well, at least for the purposes of this post) business process. We have business processes in SAP that have workflows behind them and use various means of interaction to keep it moving (i.e. nag-mails, etc.). We have similar workflows in SharePoint for document approval processes and the like. We also have a third set of workflows (business processes) that are either not automated at all, or not in any sort of consistent interface. It is this last set processes that I'm trying to "fix".

I have a couple of "rules":

1. The interface must live in SharePoint (at least the end-user facing portion)
2. The "workflow" aspects of the system should utilize one of the two existing execution engines we have running (SAP or WF in MOSS)
3. The workflow designer should be comfortable for a business analyst to use, and preferably an extension to a tool they are already using (i.e. Visio add-in).
4. The workflow designer should be able to map actual requirements to the steps in the process (where applicable) and serve as a documentation source of sorts.
5. The process execution system must provide a webparts (or at least the ability to expose the data as webparts) for the following scenarios:
   1. Overall health of the system
   2. Current user's workflows currently in action
   3. Visual representation of a given workflow process
   4. Analysis relative to the execution stats for each workflow instance and type
      1. i.e. X step of workflow Y is over the planned duration without having completed. Some indication should exist that this process is out of line
      2. i.e. Workflows of type X average Y days with the minimum being Z days and the maximum being T days. (and trends over time)
6. There must be a hard link between process definition and process execution. The File | Print approach is not sufficient
7. The system should be standards-based. Meaning, I should be able to import/export the workflow definitions (at least at some level of granularity) to an industry standard such as BPEL or BPMN in order to be able to share or compare that process with other organizations.
8. The system should be able to provide governance over the models.
9. In all aspects possible, the system should provide a consistent story WRT the development paradigm we have selected (MSFT .NET, ASP.NET, Windows Workflow, SharePoint, etc.).

Reality...

1. I'm not exactly certain I'm ready to back down on any of the items above, however I'm coming to the conclusion that there may be a need for a "third" execution engine - meaning, all of the vendors that I saw that had platforms that did what I wanted, either used their own custom execution engine or hosted their own instance of WF separate from MOSS. Even MSFT pushed BizTalk as the "main" process execution engine for *serious* workflows.
2. If reality point #1 is in fact true, the process execution platform should be based on WF allowing the developers to have a consistent development paradigm.

MSFT/BPM/SOA Session 6

The sixth session I attended was entitled "Next Generation Business Applications" and was presented by Mike Walker of the Architecture Strategy Team (MSFT).

This presentation dove-tailed nicely with the one I had attended earlier that day on the human aspects of SOA/BPM. The real gist of this session was to challenge the user to reconsider the requirement to write distinct user interfaces for each of the applications he/she is tasked to build. In much the same vein as the earlier talk, he was encouraging us to think of ways to expose the user-facing aspects of our applications in the context of applications that the user is already familiar with, such as using the OBA approach and embedding the application in a common productivity tool such as Word, Excel, or PowerPoint. The point being, that the less you force you user to "learn" in order to use your application, the more likely they are to use it. Further, the user doesn't really give a hoot if your application is fully SOA compliant and uses your best-of-breed enterprise service bus, and blah, blah, blah... They care about, does it work. What do I have to do in order to do my job. Tell me that, let me do that, and then leave me alone. and... that's not altogether a bad view of things.

The point is not that OBAs, or "Composite applications" are the only way to go, but rather that it should definitely be on our "radar" and should be considered (at least) prior to beginning to build a full UI.  Another take-away from this session was the fact that the more consistency you can drive in your applications (or, more importantly across your applications), the better off you are. Building a "unique" or "fully custom" user experience is not likely a good thing.

MSFT/BPM/SOA Session 5

This session was titled: "The Human-Facing SOA: Realizing the Value of Web Services with Microsoft Office SharePoint Server 2007" presented by Hugh Taylor (Senior Marketing Manager, MSFT)

So, reading the title of this session, I assumed it was a technical presentation focused on SharePoint web services.... however, that was not the case. Rather, it was a business-focused presentation that centered completely on the "users" of any given SOA. While not being exactly what I expected, it was a very good session and well worth the time to sit through it.

As with the other sessions, it's a little easier for me to simply comment on poignant comments from the speaker than to attempt a proper summary, so here goes:

• People are often the missing ingredient for SOA business value
• If you are trying to measure the ROI of your SOA project, you'll likely be disappointed. Some common reasons are:
  • slower than anticipated uptake
  • training takes more time than planned
  • various Governance and security hang-ups
• One of the factors in all of this is a lack of a familiar interface
• We, as technologists and system designers MUST learn to focus on the user, rather than the technology. Hugh Macleod has a comic that aptly puts what I want to say here:
  
  
• We seem to get so wrapped around the axle with respect to how the system interacts with itself or the technology deployed within the system that we fail to realize that without a user, our system is useless.
• He stressed the importance of accurate communication at all levels of the SDLC. Often business drivers and technologists are "talking past" each other.
• He noted that as software developers and coders we often underestimate the power of habit, organizational norms, etc. Just because it's "better" doesn't mean that it's "better".

At this point he started talking specifically about the MSFT platform and the talk became less interesting to me :)

All in all, one of the best, most thought-provoking sessions I attended.

MSFT/BPM/SOA Session 4

The fourth session I attended was titled "BPM Best of Breed: Accenture, Avanade, IDS Scheer, and Microsoft - Delivering Closed Loop BPM and was presented by Brian Wilkinson (Accenture), Altan Enginaleve (IDS Scheer) and Dana Kaufman (Microsoft).

The objective of this session was to do a "deep-dive" into the demo that these same folks presented during the keynote earlier that day (which was very interesting).  What I had feared was the case, worked out to be, in fact, reality - the demo was a bit more duct-tapped together than I had hoped. The intent of the project was to illustrate to us and their customers what could be done ("paint the picture" if you will).

I have to admit that the ARIS platform was very intuitive and interesting, although I found myself a little turned off by the extensive use of Java and JDBC connections (I know, I know, but I bleed blue...). As with many of the sessions, the speakers made some general comments/points that I thought were interesting...

• Many organizations have very well documented business processes, however, those rarely reflect reality (no direct impact on execution)
• They (as consultants and based on their experience) encourage people to avoid "as-is" process modeling and focus more on "target" process modeling. In their experience, it is not uncommon to find that upwards of 80% of the "process" that existed was not tied to true requirements, but rather historical, or "just because that's how we do it" reasons. Simply focusing on what *really* needs to happen (almost ignoring for a little while how it is currently being done) can be a great source of improvement to efficiency.
• As often is the case, the starting point is processes - processes can be documented, metrics can be understood - this is not a technology problem. These (both processes and the metrics that you want to collect about those processes) must be worked out prior to any "silver technology bullet" "fixing" things.

Think Big, Start Small, Scale Fast.

MSFT/BPM/SOA Session 3

The third session I attended was a lunch session sponsored by K2 (http://www.k2.com). I was looking forward to this presentation as I've heard quite a bit about K2, and they've been doing workflow in SharePoint long before MSFT was.

During this session, they focused on their "Black Perl" product which is the current release. Based on the demo, a quick summary of the product is as follows:

- Middleware platform that provides workflow and reporting based on the concept of "business entities"

- They have a concept of a "SmartObject" which is mechanism for creating, what I would call a composite object/view of a singe entity within the organization regardless of the data source. This means, I could create a smartobject which represents an "employee", for which some of its attributes/values are derived directly from SAP, others from AD, and maybe still others in a custom built SQL DB. This object abstracts for the user all of the complexities of interacting with the data sources on the backend. In the demo, the showed creating a smart object for which the data resided in SAP and then exposed that "entity" in a number of different places (SSRS report, workflow, etc.).

All in all, the demo was pretty interesting and there is some applicability to the problem set I'm currently working with, but I had a hard time imagining a non-developer using their "non-developer" focused tools. I'm certain that it would come with time, but seemed like a higher barrier to entry than I was looking for.

That being said, as mentioned before, their demo focused specifically on the Black Perl product and, having spent some more time on their web site since, there are clearly other products/updates to existing products in the works that might make the entire platform much more of a "fit" for the scenario we are trying to fix.

MSFT/BPM Session 2

The section session I attended was entitled "People-Read Processes with SharePoint Workflows and Forms Services" and was presented by Christian Stark, a Senior Product Manager for Microsoft.

The presenter gave a survey at the beginning and hinted that, if you met a certain criteria (which I did) that you may not want to sit through the session as it would likely be familiar material. In retrospect, I should have listened and looked for another session. The presentation wasn't wasted, just not full of "new" material.

Christian divided the session into two main sections - the first of which was slides and discussion while the second half focused on demos.

During the theory section, he mentioned a few things that seemed to resonate with me (presented below in no particular order):

• Most end-users want to "do their job" without being "encumbered" (or, at least not overly so) with processes
• Most IT users have the exact opposite desire... we want to lock users into a very specific process so we get accurate, consistent data and can properly execute the workflow
• A successful implementation will be (at least partially) characterized by:
  • IT successfully exposing the workflow endpoints directly in interfaces that the end users are already comfortable with (i.e. MS Word, Excel, PowerPoint).
  • Making the processes task-driven rather than email-driven --> takeaway: if you are going to send nag mails as part of the workflow, make sure that the email is directly actionable (i.e. includes a link, form, or button against which the user can immediately execute).

Objectives/Responsibilities on IT

• Simplify the human interactivity aspects of the workflow
• Simplify the data collection
• Simplify the participation
• Simplify the process management and monitoring

MSFT/BPM Session 1

Today started out with the keynote for the conference and it was certainly not disappointing. The following is basically a brain dump of the information presented along with some commentary.

Dr. Donald Ferguson (previously a "Fellow" at IBM) opened the conference by explaining talking a little bit about why he left IBM to join MSFT and explaining some of the MSFT vision for workflow and process modeling. Some interesting points from his portion of the talk:

• Tools too often tend to speak in infrastructure/technical terms without the proper level of abstraction... this is a trend that needs to change (i.e. we as technicians need to adapt both ourselves and the tools we expose to end users to speak a language that they understand).
• Postulated that all organizations exercise at least some level of model-driven development. His position is that even if you don't have a formal process, you likely have informal processes (i.e. white-boarding)
• Need to automate as much as possible.
• Automation is *not* specifically a threat to people, it is more that it frees them to focus on higher-value work
• Discussed the Enterprise Service Bus which, ideally, organizations will have already deployed.
• Discussed the Internet service bus - which, is a ESB focused on the web as a unit. This is a service that will be offered/hosted by MSFT
• The goal is to make it simpler for other services/companies/vendors/etc. to connect to this "web of services" (in contrast to the current perception of the "web of pages")

Robert Wahbe (Corporate VP, MSFT) got up and began giving a pretty interesting talk on the MSFT stack that supports SOA/BPM as we know it today.  Some general points from his section of the discussion:

• We (as an industry) are looking at new application styles
  • We need to think more about cross-boundary (i.e. firewall) rather than only within the firewall
  • We need to think about federation, rather than domain-specific
  • We need to think about Services rather than objects
• We face a number of challenges...
  • this new model of development forces the integration of business and IT (a problem that should really be addressed currently, but this new approach will force it
  • Model-driven development will require improvements and structure ion your software development lifecycle
  • Increasingly need to deal with heterogeneous platforms
  • all of this, while retaining (or creating) flexibility and adaptability
• The MSFT Approach
  • Current product stack - BTS, .NET, VS, Office, SQL, SC, Dynamics
  • Good support for services (WCF, BTS)
  • Gave a demo of a collaborative project of Accenture, Avanade, IDS Scheer, and Microsoft... some pretty interesting functionality but not yet fully baked.
  • There is guidance on ESB available at http://www.codeplex.com/esb
• General...
  • Favorite Quote: "You gave me exactly what I asked for, but not what I wanted" - common symptom of the disconnect b/t business personnel and development staff
  • the services and work we write needs to be reusable.
  • The documentation of the business process is as important as the process itself

The next wave... this is where things got really interesting.

MSFT has the goal of making it *all* model driven. They announced a platform code-named "Oslo" which is a collection of existing products lines (but future versions) (BTS 6, BizTalk Services 1, VS 10, System Center, .NET Framework 4) as a "new way to build applications".

This platform includes a new, generic modeling tool that is available across all disciplines (non domain specific). This tool allows for everyone in the application chain to perform their modeling in the same tool, link the models together, and also provide a repository for those models. They also indicated that the universal tool was not the exclusive tool for these models, but rather you could use domain-specific tools, store them in the repository, and then use this tool to link things together and view the "big picture". What was particularly interesting was the notion of being able to link everything from the initial requirements modeling, through to system architecture to design, to coding, through deployment and eventually to monitoring and maintenance all within the same tool... the ability to link a specific performance problem or improvement directly back to the requirement(s) that caused it is very interesting.

They then spent some additional time discussing the ISB... specifically on the features that allow you to control which services are hosted in the cloud vs. those that are hosted locally to your enterprise... It will be interesting to see how this plays out in the long term... meaning, the idea is interesting, but it requires (at least from my current understanding) you to "trust" MSFT to host and maintain the service... which then begs questions relative to security, reliability, etc. of the platform and, of course, if you are running a business based on these, the SLAs in place and how one deals with a breech or service outage.

Interesting links so far:

• http://www.codeplex.com/esb
• http://k2.net/
• http://www.microsoft.com/biztalk/solutions/bpm/bpapartners.mspx
• http://www.biztalk.net/
• http://www.skelta.com/
• http://www.ids-scheer.com/index.html
• http://www.pnmsoft.com/

Microsoft's Fifth Annual SOA and BPM Conference

I'm in Redmond this week attending Microsoft's fifth annual SOA and BPM conference. It's pretty interesting so far, and I'm intending on writing a number of different posts (one per session that I attend). For the last nine months I've been working in the federal IT arena stationed working with a reasonably large organization. One of our current objectives is to evaluate means of improving the governance of our processes both at the definition as well as the execution level. The means, that I'm interested in this conference specifically from the aspect of understanding where MSFT is going in the SOA space but also where the various partners play in the space, and working to understand how the various tools work together to affect a workable solution.