Todd Biske: Outside the Box

Archive for the ‘SOA’ Category

For those of you attending Oracle OpenWorld, please come to my session on Monday the 12th at 4:00pm in the Golden Gate 3 room of the Hilton Hotel. I will be participating in a panel discussion on SOA Governance best practices. Based on the pre-call sessions I had with the other panelists, it should be a very informative session. While you’re at the conference, stop by the conference bookstore and pick up a copy of my book. Thanks to my publisher for making it available.

In a two-part podcast series, Dave Berry from Oracle’s Fusion Middleware team and Mike van Alst, a consultant with IT-eye, discussed some remarks I made in an earlier OTN Arch2Arch podcast regarding SOA and reuse. Specifically, I tried to de-emphasize the reuse aspect of SOA. Many reuse programs that I’ve seen or read about have two key elements:

1. Building things in a reusable manner
2. Making those things visible

While noble goals, these approaches are at significant risk of producing the intended results. The first item has a fundamental problem in that it is all but impossible to define exact what “building in a reusable manner” is. We can use open, interoperable standards rather than closed, proprietary ones, but is this the key barrier to reuse? There’s probably some low hanging fruit that this will capture, but there’s so much more to reuse than this. From a technical standpoint, one must also consider the structures of the information being exchanged and the varying granularity of the information being exchanged, among other things.

On the second item, visibility is important, there’s no doubt about it. But visibility without context will not be successful. It’s a matter of providing the right information at the right time. Too many initiatives that are associated with the collection of IT artifacts, be it reuse, SOA, portfolio management, ITSM, or any of the like, fail because the information is never put into the context of the processes that need that information. How many times have you seen the information collected as part of a fire drill for an immediate need, only to grow stale once that fire drill is completed.

The two things I recommend are service ownership and linkage to key IT processes. If you’ve heard me talk on panel discussions at conferences, you’ll know that my answer to the question, “What’s the one piece of advice you have for companies adopting SOA?” has always been, “Define your service owners.” Someone is given the responsibility for a functional area, providing capabilities to the rest of the organization and accountable for driving out the redundancies that may exist. This is a tricky exercise, because service ownership has a cost associated with it. Expending that cost for a service that is only used by one consumer can lead to waste, so it’s not a silver bullet. It does, however, being the cultural change from a project-driven organization to more of a product-driven/service-driven organization. Without having someone accountable for the elimination of redundancy in a domain and serving the needs of consumers, it won’t happen.

The second piece of advice is the process integration. To avoid creating repositories that see infrequent use after initial population, you have to define the role of that information in the IT processes. If you have a service repository, when do you expect project architects and designers to look into that repository for services that may be appropriate. How about it the strategic planning process? The scoping effort for a project likely begins long before a project architect is assigned? How is the service repository used in those activities? By defining the links with key IT processes and ensuring that those processes are changed to use the repositories involved, with appropriate governance to make sure those changes are occurring, you will make sure that your services are visible, and more importantly, that the right people are looking for them at the right time.

Jeff Schneider posted the following comment to my previous post, “What are your EA Services?”

Your services feel too ‘responsive’ – like someone has to pick up the phone and call you in order for EA to be valuable. Thoughts?

I was hoping someone would ask a question along these lines, because it’s something I’ve thought a lot about. Services, by their nature, should seem very ‘responsive.’ After all, there should always be a service consumer and a service provider. The consumer asks, the provider does. In the context of an enterprise architecture team, or any other team for that matter, you do have to ask the question, “If no one is asking me to do this, why am I doing it?” Is that stance too theoretical, though?

When I think about this, I think services in this context (ITIL/ITSM) follow the same pattern that can be used for web services. There are services that are explicitly invoked at the request of a consumer, and then there are services that are executed in response to some event. In the latter case, the team providing the service is the one monitoring for the event. If some other team was monitoring for it, and then told your team to do something, then we’re back to the request/response style with one team acting as consumer and the other acting as the provider.

Coming back to the EA services, I think an Enterprise Architecture team will typically have a mix of request/response-style and event-driven services. It’s probably also true that if the EA team is closer to project activity, you’ll see more request/response services done on behalf of project teams, such as the Architectural Assessment Services and the Architectural Consulting Services I mentioned. If your EA team is more involved with portfolio management and strategic planning, then it’s possible that you may have more event-driven services, although these could still be request/response. Strategic direction could be driven by senior management, with direction handed down on areas of research. That direction represents a service request from that strategic planning group to the EA team. In an event-driven scenario, the EA team would be watching industry trends, metrics, and other things and making their own call on areas for research. It’s a subtle difference.

Anyway, I do think Jeff has a valid point, and as part of defining the services, you should classify the triggers that cause the service to be invoked. This will clearly capture whether they are event-driven or request/response. If your list is entirely request/repsonse, that’s probably something to look into. That may be what’s needed today, but you should be practicing good service management and planning for some event-driven services in the future if your team continues to provide value in the organization.

Thanks to Jeff Adkins for the good discussion about this on Twitter.

A week or so ago, I asked about defining on EA services on Twitter. My use of the term services here is in more of the ITIL/ITSM sense, not what typically comes to mind when discussing SOA, but I could have another blog post just dedicated to that subject. I’ve been working to define EA services at work, and it’s been a very interesting exercise, and I had hoped that other EA’s (or former EA’s) on Twitter would have something to contribute.

Something that is a bit surprising to me, is that many IT teams struggle to explain exactly what they do, especially those whose primary purpose isn’t project work. This doesn’t mean that the team isn’t needed, but it does put you at risk of having the team be defined by the individuals rather than by their responsibilities. Depending on who the individual is, you get a different set of capabilities, making it difficult to quantify and measure what the team, rather than the individual does. A conversation with my boss and another team member on a different subject brought up the term “define by example” and we all agreed that it’s usually a bad thing. Examples are very important in illustrating the concept, but they shouldn’t be the definition. The same thing goes for a team. Your team should not by defined by individuals on it, but rather the individuals on it should be providing the defined services.

Getting back to the subject, the initial list of EA services I came up with, and vetted by Aleks Buterman, Leo de Sousa, and Brenda Michelson are:

• Architectural Assessment Services: The operations in this service include anything that falls into the review/approve/comment category, whether required or requested. Ad hoc architectural questions probably go here, but those are one that I’m still sitting on the fence about.
• Architectural Consulting Services: The operations in this service include anything where a member of the EA team is allocated to a project as a member of that project team, typically as a project architect. The day-to-day activities of that person would now be managed by a project manager, at least to the extent of the allocation.
• Architectural Research Services: The operations in this service are those that fall into the research category, whether formal or informal. This would include vendor conversations, reading analyst reports, case study reviews, participation in consortiums, etc.
• Architectural Reference Services: The operations in this service are those that entail the creation of reference material used for prescriptive guidance of activities outside of the EA team, such as patterns, reference models, reference architectures, etc.
• Architectural Standards Services: Very similar to reference services, this service is about the creation of official standards. I’m still on the fence as to whether or not this should be collapsed into a single service with the reference services. Sometimes, standards are treated differently than other reference material, so I’m leaving it as its own service for now.
• Architectural Strategy Services: Finally, strategy services capture the role of architecture in strategy development, such as the development of to-be architectures. If there is a separate strategy development process at your organization, this one represents the role of enterprise architecture in that process.

Now, the most interesting part of this process has not been coming up with this list, but thinking about the metadata that should be included for each of these services. Thinking like a developer, what are the inputs and outputs of each? Who can request them? Are any internal services (e.g. always requested by the EA manager) only, and which ones are external services (e.g. requested by someone outside of EA)? What are the processes behind these services? Are these services always part of a certain parent process, or are they “operations” in multiple processes? How do we measure these services? You can see why this suddenly feels very much like ITIL/ITSM, but it also has parallels to how we should think about services in the SOA sense, too. Thinking in the long term, all of these services need to be managed. What percentage of work falls into each bucket? Today, there may be a stronger need to establish solid project architecture, leading to a higher percentage of time spent consulting. Next year, it may shift to strategy services or some other category. The year after that, the service definitions themselves may need to be adjusted to account for a shift toward more business architecture and less technology architecture. Adjusting to the winds of change is what service management is all about.

So, my question to my readers is, what are your EA services? I’m sure I’m not the only EA out there who’s had to think about this. Even if your EA organization hasn’t, the next time you fill out your time card, think about what “service bucket” your efforts fall into. Do my categories make sense for what you do each week or month? If not, what’s missing. If it’s unclear which bucket something should go in, how would you redefine them? A consistent set of EA service definitions can definitely help all of us.

I just read the Forrester document, “Inquiry Spotlight: Building An EA Practice, Q2 2009,” written by Gene Leganza and Katie Smillie, with Alex Cullen and Matt Czarnecki, and there’s one line that really stuck out for me.

When creating EA artifacts, you should focus on “building what sells” more than on “selling what you build.”

I think I should take that line and make a poster out of it. This consumer-first thinking is really, really important. I’ve seen too many things that were written for the convenience of the author rather than the consumer, and it never is as successful as it should be. This applies to EA artifacts, to user interfaces, to services, and just about anything else that’s supposed to be consumed by someone other than the person who wrote it. If you don’t make it easy to consume by the intended audience, they won’t consume it at the rate you desire. The trap that you can also fall into is to fail to recognize that you have more than one audience. If you assume a single, broad audience, then you wind up with a least common denominator approach that frequently provides too little to be useful to anyone. While some “100-level” communication is a good thing, it must be followed up with the “200-level” and “300-level” communication that is targeted at particular audiences and particular roles. For example, if you’re planning your communication around an enterprise SOA strategy, you may create some 100-level communication that is broad enough to wet the appetite of project managers, organizational managers, and developers, but not enough to tell any of them how it will impact them in detail. Follow that up with pointed conversations targeted specifically at the role of the project manager, the organizational manager, and the developer to get the messages across to them, and them only.

Finally, getting back to EA artifacts, consider not just the roles, but also the context in which the artifacts will be used. If the artifacts are used in project activities, then structuring them so the appropriate information is provided in the appropriate phase of the project is a good thing. Organizing the artifact to where people must hunt all over for the information they need at a particular point in time is not a good thing.

Once again, take Gene and Katie’s words to heart: Focus on building what sells more than selling what you build.

Thanks to Google alerts, I found out that my book, SOA Governance, is now available via Safari Books Online. You can access it here. If you enjoy it, consider voting for me as the Packt Author of the Year.

The publisher of my book, Packt Publishing, has announced a competition for Author of the Year. You can find out more about the award here, as well as cast your vote. I’ll be perfectly transparent and state that there is a cash award associated with this, although I’d be posting this even if there wasn’t. I’m proud of the book that I wrote and if others have received value from it, that makes me even happier. If you feel so inclined to recommend me to my publisher, I’d be honored, but know that I’m already honored by the fact that you’ve either read or just considered reading my book. Packt is also giving away some prizes to random voters, so there may be something in it for you, too. Thanks for your consideration, and hopefully, your vote!

I’ve been going through the TOGAF 9 documentation, and in the Application Software section of the Technical Reference Model, there are two categories that are recognized, Business Applications and Infrastructure Applications. They define these two as follows:

Business applications … implement business processes for a particular enterprise or vertical industry. The internal structure of business applications relates closely to the specific application software configuration selected by an organization.
Infrastructure applications … provide general purpose business functionality, based on infrastructure services.

There’s a lot more to the descriptions than this, but what jumped out at me was the typical black and white breakdown of infrastructure and “not” infrastructure. Normally, it’s application and infrastructure, but since TOGAF uses the term infrastructure application, that obviously won’t work, but you get the point. What I’ve found at the organizations I’ve worked with is that there’s always a desire to draw a black and white line between the world of infrastructure and the application world. In reality, it’s not that easy to draw such a line, because it’s an ever-changing continuum. It’s far easier to see from the infrastructure side, where infrastructure used to mean physical devices, but now clearly involves software solutions ranging from application servers to, as TOGAF 9 correctly calls out in their description of infrastructure applications, commercial of the shelf products.

The biggest challenge in the whole infrastructure/application continuum is knowing when to shift your thinking from coding to configuration. As things become more commoditized and more like infrastructure, your thinking has to shift to that of configuration. If you continue with a coding and customization mentality, you’re likely investing significant resources into an area without much potential for payback. There are parallels between this thinking and the cloud computing and software as a service movements. You should use this thinking when making decisions on where to leverage these technologies and techniques. If you haven’t changed your thinking from coding to configuration, it’s unlikely that you’re going to be able to effectively evaluate SaaS or cloud providers. When things are offered as a service, your interactions with them are going to be a configuration activity based upon the interfaces exposed, and it’s very unlikely that any interface will have as much flexibility as a programming language. If you make good decisions on where things should be configured rather than coded, you’ll be in good shape.

People who have worked with me know that I’m somewhat passionate about having a well-defined engagement model. More often than not, I think we’ve created challenges for ourselves due to poorly-defined engagement models. The engagement model normally consists of “Talk to Person A” or “Talk to Team B” which means that you’re going to get a different result every time. It also means that interaction is going to be different, because no one is going to come to Person A or Team B with the same set of information, so the engagement is likely to evolve over time. In some cases, this is fine. If you’re part of your engagement model is to provide mentoring in a particular domain, then you need to recognize that the structure of the engagement will likely be time-based rather than information-based, at least in terms of the cost. Think of it as the difference between a fixed-cost standard offering, and a variable cost (usually based on time) from a consulting firm. I frequently recommend that teams try to express their service offerings in this manner, especially when involved in the project estimation process. Define what services should be fixed cost and define what services are variable cost, and what that variance depends on. This should be part of the process for operationalizing a service, and someone should be reviewing the team’s effort to make sure they’ve thought about these concerns.

When thinking about your services in the ITSM sense, it’s good to create a well-defined interface, just as we do in the web service sense. Think about how other teams will interact with your services. In some cases, it may be an asynchronous interaction via artifacts. An EA team may produce reference models, patterns, etc. for other teams to use in their projects. These artifacts are designed in their own timeline, separate from any project, and projects can access them at will. Requests to update them based on new information go into a queue and are executed according to the priority of the EA manager. On the other hand, an architecture review is executed synchronously, with a project team making a request for a review, provided they have the required inputs (an architectural specification, in most cases), with the output being the recommendations of the reviewer, and possibly a formal approval to proceed forward (or not).

If you’re providing infrastructure services, such as new servers, or configuration of load balancers, etc., in addition to the project-based interactions, you must also think about what your services are at run-time. While most teams include troubleshooting services, sometimes the interface is lacking definition. In addition, run-time services need to go beyond troubleshooting. When the dashboard lights are all green, what services do you provide? Do you provide reports to the customers of your services? There’s a wealth of information to be learned by observing the behavior of the system when things are going well, and that information can lead to service improvements, whether yours or someone else’s. Think about this when you’re defining your service.

Well, I have to hold true to my bet with Leo de Sousa. Congratulations to the Vancouver Canucks on sweeping my St. Louis Blues in the first round of the playoffs. They actually played very well, and I’d rather see them win than either Detroit or San Jose, so go Canucks. Now I can go back to paying attention to the St. Louis Cardinals (who came back from 4-0 down today to beat the Mets 6-4), and thankfully, Vancouver doesn’t have a major league baseball team.

I had a brief conversation with Nick Gall (Twitter: ironick) of Gartner on Twitter regarding designing for change. Back in the early days of SOA, I’m pretty sure that I first heard the phrase, “we need to build things to change” from a Gartner analyst, although I don’t recall which one. Since that time, there’s been a lot of discussion on the subject of designing/building for change, usually tied to a discussion on REST versus WS-*. Yesterday, I stepped back from the debate and thought, “Can we ever design for change, and is that really the right problem?”

As I told Nick, technology and design choices can certain constrain the flexibility that you have. Think about the office building that many of us work in. There was a time when they weren’t big farms of cubicle and they actually had real walls and doors. Did this design work? Yes. Was it flexible enough to meet the needs of an expanding work force? No. I couldn’t easily and quickly create new conference rooms, change the size of spaces, etc. Did it meet all possible changes the company would go through? No. Did the planners ever think that every cubicle would consume the amount of electricity they do today? What about wiring for the Internet? Sometimes those buildings need to be renovated or even bulldozed. The same thing is true on the technology side. We made some design decisions that worked and were flexibility, yet not flexible enough for the change that could not have been easily predicted in most companies, such as the advent of the internet.

Maybe I’m getting wiser as I go through more of these technology changes, but for me, the fundamental problem is not the technology selection. Yes, poor design and technology selection can be limiting, but I think the bigger problem is that we have poor processes for determining what changes are definitely coming, what changes might be coming, and how and when to incorporate those changes into what IT does, despite the available predictions from the various analysts. Instead, we have a reactive, project-driven approach without any sort of portfolio planning and management expertise. To this, I’m reminded of a thought I had while sitting in a Gartner talk on application and project portfolio management a year or two ago. If I’m sitting in a similar session on service portfolio management 5 years from now, we’ve missed the boat and we still don’t get it. Develop a process for change, and it well help you make good, timely design choices. The process for change involves sound portfolio management and rationalization processes.

Fellow Twitterer Leo de Sousa posted a review of my book, SOA Governance, on his blog. Leo is an Enterprise Architect at the British Columbia Institute of Technology, and is leveraging the book on their journey in adopting SOA. Thanks for the review, Leo. I’m glad you posted it before the Stanley Cup playoffs begin as my St. Louis Blues will be taking on your Vancouver Canucks, and I wouldn’t have wanted the upcoming Blues victory to taint your review!

I recorded a podcast on various SOA Governance topics with Bob Rhubart, Cathy Lippert, and Sharon Fay of Oracle as part of Oracle’s Arch2Arch Podcast series. You can listen to part one via this link, or you can find it at Oracle’s ArchBeat site here.

On Friday, the SOA Chief (Tim Vibbert), Brenda Michelson, and I had a conversation on Twitter regarding SOA governance and anonymous service consumers. Specifically, how do you provide run-time governance for a service that is accessed anonymously?

If you’ve read this blog or my book, you’ll know that my take on run-time SOA governance is the enforcement and/or monitoring of compliance with the policies contained within the service contract. Therein lies the biggest problem: if the service consumer is anonymous, is there a contract? There’s certainly the functional interface, which is part of the contract, but there isn’t any agreement on the allowed request rates, hours of usage, etc. So what do we do?

The first thing to recognize is that while there may not be a formal contract that all consumers have agreed to, there should always be an implied contract. When two parties come to the table to establish an agreement, it’s likely that both sides comes with a contract proposal, and the final contract is a negotiation between the two. The same thing must be considered here. If someone starts using a service, they have some implicit level of service that they expect to receive. Likewise, the service provider knows both the capacity they currently can handle as well as what how they think a typical consumer will use the service. Unfortunately, these implied contracts can frequently be wrong. The advice here is that even if you are trying to lower the barrier for entry by having anonymous access, you still need to think about service contracts and design to meet some base level of availability.

The second thing to do, which may seem obvious, is to avoid anonymous access in the first place. It’s very hard to enforce anything when you don’t know where it’s coming from. Your authorization policy can simply be that you must be an authenticated user to use the service. Even in an internal setting, having some form of identity on the message, even if there are no authentication or authorization policies, becomes critical when you’re trying to understand how the systems are interacting, perform capacity planning, and especially in a troubleshooting scenario. Even services with low barriers to entry, like the Twitter API, often require identity.

The next thing you should do is leverage a platform with elasticity. That is, the available capacity should grow and shrink with the demand. If it’s anonymous, and new consumers can start using it simply by getting the URLs from someone else, you have no control over the rate at which usage will scale. If the implied level of availability is that the service is always available, you’ll need on-demand resources.

Finally, you still need to protect your systems. No request is completely anonymous, and there are things you can do to ensure the availability of your service against rogue consumers. Requests will have source IP addresses on them, so you can look for bad behavior at that level. You can still do schema validation, look for SQL injection, etc. In other words, you still need to do DoS protection. You also should be looking at the usage metrics on a frequent basis to understand the demand curve, and making decisions accordingly.

Tony Baer joined the SOA Consortium on one of its working group conference calls this week to discuss his research on connections between ITIL and SOA. Both he and Beth Gold-Bernstein have blogged about the call, Beth focusing on the broader topic of SOA and ITIL, and Tony talking about the topic of service ownership, as these topics were the meat of the conversation between Beth, Tony, and myself.

I’ve spent the past few years thinking about all things SOA, and recently, I completed the ITIL v3 Foundations certification and have been doing a lot of work in the ITIL/ITSM space. When you move away from the technology-side of the discussion and actually talk about the people and process side of the discussion, you’ll find that there are significant similarities between ITIL/ITSM adoption and SOA adoption. Tony had a diagram in his presentation that illustrated this that Beth reproduced on her blog. Having looked at this from both the SOA world of the application developer and the ITIL/ITSM world of IT operations, there’s a lot that we can learn from ITIL in our SOA adoption efforts. Foremost, ITIL defines a role of Service Manager. Anyone who’s listened to my panel discussions and heard my answer to the question, “What’s the one piece of advice you have for companies adopting SOA?” you’ll know that I always answer, “Make sure all your services have owners.” I’ve decided I like the term “Service Manager” better than “Service Owner” at this point, but if you refer to past posts of mine, you can think of these two terms synonymously.

So what does a service manager do? Let’s handle the easy one. Clearly, service management begins with the initial release of the service. The service manager is accountable for defining this release and putting the project in motion to get it out the door. This involves working with the initial service consumer(s) to go over requirements, get the interface defined, build, test, deploy, etc. Clearly, there’s probably a project manager, developers, etc. helping in the effort, but in a RACI model, it’s the service manager who has accountability. The work doesn’t end there, however. Once the service is in production, the service manager must be receiving reports on the service utilization, availability, etc. and always making sure it meets the needs of the consumer(s). In other words, they must ensure that “service” is being provided.

They must also be defining the next release of the service. How does this happen? Well, part of it comes from analysis of current usage, part of it comes from external events, such as a merger, acquisition, or new regulations, and part of it comes from seeking out new customers. Some consumers may come along on their own with new requests. Reading between the lines, however, it is very unlikely that a service manager manages only one service. It is more likely that they manage multiple services within a common domain. Even if it is one service, it’s likely that the service has multiple operations. The service manager is the one responsible for the portfolio of services and their operations, and trying to find the right balance between meeting consumer needs and keeping a maintainable code base. If there’s redundancy, the service manager is the one accountable for managing it and getting rid of it where it makes sense. This doesn’t negate the need for enterprise service portfolio management, because sometimes the redundancy may be spread across multiple service managers.

So what’s the list? Here’s a start. Add other responsibilities via comments.

• Release Management (a.k.a. Service Lifecycle Management)
• Production Monitoring
• Customer (Consumer) Management
• Service Management
• Marketing
• Domain Research: Trends associated with the service domain
• Domain-Specific Service Portfolio Management

Think hard about this, as it’s a big shift from many IT organizations today. How many organizations have their roles strictly structured around project lifecycle activities, rather than service lifecycle activities? How many organizations perform these activities even at an application level? It’s a definition change to the culture of many organizations.