Saturday, March 17, 2012

Stop B*tching About Local Optimizations

I'm kinda tired of hearing folks whine about everything under the sun being a local optimization. I don't care how much you quote Ackoff, Argyris, Senge or anyone else coz here's the deal:

Unless you can take a single action and reinvent the whole effing universe instantly then EVERYTHING you do is a local optimization!

Does this seem extreme? I don't think so, because one of the main problems with improving systems is to know exactly where the system boundaries are. For example: Is a team a system? Yes! But it operates in a department, which is also a system, which is inside an organization which is inside a community which is inside a city, inside a state, country, world, etc… So how does one begin to take action which is not some local optimization?

Here's another thing to chew on…even if we draw the boundaries of the system to just the department level, that is still (in most enterprises) a whole lot of people, interactions and influences. In order to improve a system you should first understand it. Problem is, by the time you understand the system well enough to change it, the system itself has likely changed rendering all your changes useless or non-optimal and much of your time wasted. Of course this issue explodes exponentially for each step towards a bigger system you take.

So, what are we to do? I suggest this fun little metaphor:

Act like a good Proctologist; consider the (w)hole, but treat the individuals ;)

As a change agent, whether an internal employee or an external consultant, you often have to play the cards you're dealt (or fold). You should definitely spend some initial time trying to consider as much as the system as you can. In fact, this should be something you are always doing at some level. More importantly though is to not let this paralyze you. You'll often get more data from attempting a safe-to-fail change than you will from simply sitting back trying to sense the system. Also, don't think too linearly. You can likely be influencing multiple things simultaneously.

In fact, one approach is to intentionally over optimize a local optimization. This will often make apparent to management (or even to you) where the true bottle neck in the system is. We shouldn't worry so much about doing the wrong things righter, but we should be aware that that may be the case and always work to be doing the right things.

In the end, showing improvement and building momentum can lead to exciting changes. In fairness, it can also come crashing to the ground if the right kinds of changes aren't made at some point, but this should not deter anyone who thinks something can be made better from trying to do so and it certainly should not be a reason to do nothing!

23 comments:

  1. Nice post, Matt.

    Another point to consider is that much of what coaches and change agents do is to teach people to think about these things for themselves. There is no "local optimization" for learning. It's always a help.

    ReplyDelete
    Replies
    1. Hey George, I agree whole heartedly :) If someone is just telling people what to do they are at best facilitating cargo-cult change. Teaching a village how to fish is a must-do in my opinion.

      I really cant underscore enough that changing groups of people is non-linear and that avoid sub or local optimization only makes sense in the case of a known, mechanical processes.

      Thanks for the feedback!

      -Matt

      Delete
  2. I get a sense you're responding to something left unlinked [he said, leadingly...]

    ReplyDelete
    Replies
    1. Yes and no. There was a recent post I've read, and will respond to separately, that was the final straw on "harmful local optimizations"...but I've been hearing this from all sorts for over a year (maybe more, but memory fades)

      In fact, I *used* to agree and it caused me great introspection during a few of my early engagements as an external consultant...which proved to me it's a whole bunch of codswallup :)

      Thanks for the interest in the article though...I'll be posting, and linking, to the sparking article soon.

      -Matt

      Delete
  3. Yep. Couple of things to add:

    1. Perhaps this is just a corollary to "the main problem with improving systems is know exactly where the system boundaries are", but there is also the problem of knowing whose perspective to take. Do we accept only the perspective of the guy with the "Systems Thinker" sign on the back of his jacket? It seems to me that the safest option is actually to permit/encourage change from multiple perspectives and then to let things emerge.

    2. Learning how to change is itself valuable. Best practiced locally, don't you think?

    ReplyDelete
    Replies
    1. Hi Mike,
      I guess I do do see perspective as one input to determining those boundaries, yes. Emergent change is usually my preference...even if we think we are trying something revolutionary.

      Also very much agree with learning being valuable. Safe-to-fail change is often a good approach and many times locally practiced is a good candidate for this, but not necessarily always...depends on your context and support/network.

      Thanks for the interest :)

      -Matt

      Delete
  4. Good perspective on the need to act quickly and to avoid paralysis.

    I hear what your saying, and see value in it. I also spent 30 years in a major corporation that went whole-hog into what they called "Lean" (though it wasn't really, when compared to the Lean writings of the last 50 years). By "Lean" they meant, do a boatload of small-scale kaizen projects that made a lot of individual teams "run 75% faster" etc. The enterprise never saw much overall improvement from all that activity. But it sure spent a lot on it.

    I could cite you chapter and verse, however, on many "local improvements" that screwed up other teams whose only crime was to have an unavoidable interface with the "improving team." The bigger system picture was never even considered. Optimizing at the higher level is still essential, even if defining the appropriate level of "bigger system" is ambiguous.

    It's not "either/or"...there are many Systems Engineering techniques that can get one started at least *a lot closer* to the global optima...or at least, working on a much higher-potential local optima than one would if they just started right where they found themselves when they arrived on the scene.

    You're right that many people err on the side of optimization. Experimentation is a powerful and amazing tool. I see you are using the language of complex adaptive systems...and the theory has shown us we are living in a different universe than we thought for all those years (much as Einstein showed us we didn't really live in a Newtonian universe).

    However, I've seen many err on the side of immediate action and "let the emergence sort it out" as well. My takeaway from your blog is to keep a balance between action and optimization.

    Good, thought-provoking post.

    Jim Sutton

    ReplyDelete
    Replies
    1. Hi Jim,
      I could not agree with your conclusion more! Balance is the key to most things. Intentional experimentation coupled with contextual awareness simultaneously :) and yes, I'm a big fan of complex adaptive systems…and systems thinking honestly. All knowledge, learnings, schools of thought, etc… are just more tools for a practitioner to use.

      Also, I understand your point about corporate improvement initiatives…I've been in a few of those rodeos myself (usually as the clown). But the kind of local improvements I'm thinking of (improving a team's dev practices, improving middle management processes, etc…) I don't think are a waste. Plus, as George pointed out above, a good change agent is also helping people learn how to think about the problems, not just solving them. To me this never seems wasteful, no matter the level.

      Also, one man's global optimum is another man's local. We could argue that we should optimize for an organization's bottom line. However, in the eyes of many, simply seeking to improve the bottom line or shareholder value is also a local or sub optimization.

      Thanks for your input. I like the points you raise :)

      -Matt

      Delete
    2. And likewise to your thinking. It stretched me and made an enjoyable read. (Even if I somehow embarrassingly used the plural "optima" when I meant the singular "optimum", and "your" for "you're!" I really did know better. Really. :-) Another proof that editing is always called for before hitting "send").

      All the best,

      Jim

      Delete
  5. Sometimes people advising against local optimization just use it as an excuse so they themselves don't have to change. If the change happens nonetheless, they will eventually come around because hiding isn't possible anymore.

    In a recent case - and you Matt know what I'm referring to - we are seeing positive effects of local optimization. We are helping just the programmers on several teams do something better and suddenly there is interest from the analyst in Cucumber scenarios. The same analyst that initially were not interested at all in more collaboration. It was seen as wrong to introduce automated acceptance testing just to programmers instead of truly the whole team. But doing so seems to help to break the ice somehow. Must be something about human behavior.

    ReplyDelete
  6. "Ackoff, Argyris, Senge" -- who?

    ReplyDelete
  7. From my point of view, this is one perspective on a complex issue.

    Another aspect of that issue is: you have to choose how you are going to address a problem. Sometimes the technical issues become radically simpler if you push back some elements into the business requirements.

    But, of course, doing that requires dealing with people (an issue which other comments here have raised).

    And dealing with people has some odd failure modes. The "btching" thing mentioned here is a symptom of some of them.

    So... yes... local optimizations can be good. And tackling the problem at a higher level of abstraction can also be good.

    One way of choosing between good choices involves cost/benefit analysis. Another way is to just make arbitrary decisions (especially when the apparent cost of cost/benefit analysis exceeds the change in benefit between any of the choices).

    Ultimately, though, simplicity is a virtue. When things are simple enough, even when you fail you can fail gracefully, learn from your mistakes and move on.

    ReplyDelete
  8. local peaks (optimization) have two kind of benefits:

    1 - if (when) the flood comes you'd better stand at a higher level
    2 - from a higher level you may see (and try to reach) higher peaks whose view could otherwise be blocked

    unfortunately on some occasions you could get stuck at a local peak...

    @mgaewsj

    ReplyDelete
    Replies
    1. Maybe this is a corollary to your #2, but maybe sometimes the bottom of the pit your in is so deep that you cant tell the different between lower and higher peaks :) From that vantage, they seem equally high. Not until you ascend one peak will you be able to realize another is actually higher yet.

      Delete
  9. I totally agree with George's point, but I dont agree with the post as it regards the boundaries of the system. The customer is the boundary of the system and the most important part of it.

    I strongly believe that if we can make the whole system visible to the organization and allow the teams to visualize those impediments, while empowering them to address them, there will be organic process improvement. I do not perceive my job as consultant to come in and tell you what to do. It's my job to ask you what you want and help teach you ways to see what you've got. Hopefully I can help you formulate your own ideas to help you close the gap between those two things, but saying that systems thinking is invalid because we'd have to include "The World" in our system boundary seems flawed to me.

    When I talk about systems I'm talking about a value stream from a business to a customer. I don't think anybody is advocating analysis paralysis, or doing nothing. I think helping people learn to learn, and teaching them to ask those questions themselves and adopt a culture of continuous improvement is what it's all about. "The team" may indeed be a system, but it's interconnected with other systems to the same end. I don't mean generating income so we can reproduce and keep the species alive. I mean generating value to customers and as a byproduct of that value revenue for the business. I see a "local optimization" as one that does not improve the whole system and/or does not create actual customer value.

    I'm wrong all the time, and may be wrong here. I'll be glad to hear it, if I am. :-)

    ReplyDelete
    Replies
    1. Hey Adam,
      Thanks for the interest.

      I like what seems to be your approach to consulting. I agree that it's not about telling people what to do , and I strongly agree a huge part of it is about teaching people how to learn and help create a improved culture.

      A few things though:

      First, I never said systems thinking is invalid. In fact I *did* state specifically that I like systems thinking, but it has its limitations as does any model. It's certainly not complete and as such is simply another tool in the tool belt. ANd my point wasn't that the only complete system is the universe in its entirety, but instead simply trying to illustrate the point that boundaries must be set to consider anything a system, and those boundaries are very subjective…there is no right or wrong.

      Next, I believe saying the customer is *the* boundary /and/ the most important one is a bit naive and not just a little too linear for my taste. Being the customer, they are obviously a very important part of the system, but I don't think I could every say more important than the employees, the owners, the community and the many other stakeholders involved in most organizations.

      Further, I find the definition of a value stream being simply from biz to customer a bit limiting. As stated above, there are many stakeholders and therefor many ways an organization can provide many kinds of value to each…any one of those could be considered a "stream", but they are all flowing simultaneously and sometimes will compete with one another. Ultimately it's about the actors or agents in a system understanding what is important to them, understanding their ecosystem and finding or growing a balance that works.

      I also agree with your concept of a local optimization…I mean its pretty much the text book definition. However I'm also advocating that many things that *are* local optimizations are still worth doing. For example, teaching coders how to code better or testers how to test better is a local optimization and does not add customer value if the team is working on the wrong stuff. As a consultant though, if I have the opportunity to help those individuals I will certainly take it. I'm improving them as practitioners, the development community in general, and helping the org prepare for when/if they can figure out how to do what customers really want. I also constantly seek opportunities to expand that influence and improve more and more throughout the org. It's also worth pointing out that sometimes it's unclear when a better optimization could be made until you over optimize some other subcomponent in the system.

      Remember, these are people systems. They are complex and adaptive. Much of systems thinking is a useful, but only considering one or two threads in the system (customers or value streams) can actually prevent one from seeing the whole. Balance is key. Embrace the duality :)

      Cheers,
      -Matt

      Delete
  10. So, what to do when you are "encouraged" to optimize your system (team), and are of course "empowered" to make these optimizations, yet only the minute and least significant changes are agreed upon by those that hold the weight of the team's voice? Imagine being dropped into a large forest at night, knowing only that the safest town is east, and when you dig out a compass and flashlight, your...uh...scout leader chucks the compass into the river. "Why the hell do we need that? It'll only slow us down. I'm sure the flashlight will be enough. I've used a compass in the past and it was broken so it didn't help us."

    How do you convince your "scout leader" that they just didn't know how to use the compass before without making them feel stupid, and thus jeopardizing your ability to influence change even more?

    ReplyDelete
    Replies
    1. Hey Matt,
      I think it's hard to convince your "scout master" in the environment you've described. I think I mentioned above needing a safe-to-fail environment. I think this is especially true when you are trying to influence "up". Given the situation you described I would try to continue to influence your scout master with other tools and facts. I would also focus on the other scouts, ensuring they were able to practice things that would keep them safe and help them learn in such an environment :)

      Thanks for the fun metaphor.

      Cheers,
      -Matt

      Delete
  11. Interesting viewpoint which is difficult to argue against!

    I think the problem with local optimisation is when it is performed in complete ignorance of other parts of the system (or universe), sometimes unintentionally but I suspect more often wilfully, as in "I'm only interested in my team or my department. The rest can look after themselves!"

    At least accept that there are external interactions that might be worth considering when you make your local improvements. It's a first step that may actually reap some benefits!

    Ally.

    ReplyDelete
  12. Nice post. I learn something more challenging on different blogs everyday. It will always be stimulating to read content from other writers and practice a little something from their store. I’d prefer to use some with the content on my blog whether you don’t mind. Naturally I’ll give you a link on your web blog. Thanks for sharing.

    ReplyDelete
    Replies
    1. I never mind if folks use/share my content (which is why I license it under creative commons)...just give appropriate attribution (link is fine).

      Thanks!
      -Matt

      Delete
  13. I am very pleased to find this blog. I want to thank for your time for this wonderful read!!! Keep Sharing, I'll surely be looking for more.

    ReplyDelete