Not All Lesson Learned Systems Are Created Equal

By: Todd Post, Editor of ASK Magazine

Introduction

NASA's Academy of Program and Project Leadership (APPL) has attempted a unique way of capturing and disseminating lessons learned. ASK Magazine, an APPL product, collects lessons learned in the form of stories told by managers and other project practitioners. Published bimonthly, ASK disseminates the lessons inside and out of NASA via a print publication and a web site (Reference 1).

The APPL team members working on ASK capture the lessons by inviting project managers, mostly from NASA, but also from other government agencies, industry, and sometimes academia, to tell their stories about what happened on projects, and to reflect on what they have learned while telling the story. Stories usually focus on specific subject matter: how to use prototyping as a tool to communicate better with a customer, how to let go of a popular person on a project without impacting the performance of the teammates and jeopardizing the project, or how to tailor a review so that it can be a learning experience as much as a milestone. And, as these examples show, the stories can be as various as there are issues to wrestle with on projects.

Each bimonthly issue of ASK contains approximately ten stories, and there are eighteen issues to date (June '04). The print and online versions of the magazine are intended to complement one another. The print edition, an attractive 40-page volume, has a circulation of 6,000 readers, and brings fresh lessons to them every two months.

Figure 1. ASK Magazine Lessons Learned Page (Click to Zoom)
Often the issues are focused on singular themes, and recent ones have included prototyping, reviews, project handoffs, and software project management. The online edition archives all of the back issues, making ASK lessons available to anyone with an Internet connection.

In November of 2000, the author was hired as the editor of ASK. He was not surprised then to learn that ASK was the first story-telling publication attempting to capture lessons learned. Today, however, he is surprised that it remains the only one to his knowledge. While storytelling now has several enthusiasts in the knowledge management community, there are scarcely more than a few successful examples of how it has been integrated into an organization's culture. The author knows little about the dynamics of other organizations but has thought a lot about storytelling and lessons learned and why NASA, particularly APPL, appreciates the relationship between the two as it relates to project management.

It is APPL's view that a lessons-learned system that prescribes a solution to a project management problem is flawed from the start. There is always nuance. The ASK audience astutely recognizes that no single management problem is identical to others.



What Constitutes a Lesson?

We borrow much of our thinking about lessons learned from Donald Schon, whose books The Reflective Practitioner and Educating the Reflective Practitioner lay the groundwork for our work with storytelling (Reference 2). In much broader terms than we apply to project management, Schon argues, "Reflection-inaction...is central to the art through which practitioners cope with the troublesome divergent situations of practice" (Reference 3).

And this is what stories do. They provide a space for reflection. A project manager who is challenged to bring off a deliverable on time and on budget can listen to one of his peers tell a story about similar challenges. The project manager who reflects on the story he has just heard can compare this to his own experience.

It is worth noting that no place in Schon's work does he talk about storytelling as a means of capturing lessons learned. Nevertheless, we do not believe we are skewing his message to suit our paradigm. Our subjects, like Schon's, are practitioners, each of whom "has to see on his own behalf and in his own way the relations between means and methods employed and results achieved. Nobody else can see for him, and he can't see just by being `told,' although the right kind of telling may guide his seeing and thus help him see what he needs to see" (Reference 4).

In this way, stories are another form of observation. Reading stories of how expert practitioners have solved problems requires even more from the observer. Stories demand we engage with the protagonists. By reflecting on the stories told by other project practitioners, you reframe your experience and think about it against the context of the story being told. That's what makes a story a more gratifying learning experience than many other lessons learned models--because it requires active participation.



Not Any Lesson Will Do

At NASA, lessons packaged as stories are now considered legitimate. This does not reflect, the author would argue, a predisposition towards lessons learned in general. Let us consider the evidence.

In January 2002, the GAO released a report, "NASA: Better Mechanisms Needed for Sharing Lessons Learned," which painted a discouraging picture of knowledge sharing within the agency. "Although NASA requires managers to regularly share important lessons learned from past projects through an agency-wide database," Government Executive reported shortly after the GAO report was released, "only 23 percent of managers surveyed had ever entered information into the system" (Reference 5). NASA managers explained that they neglected the database, known as the Lessons Learned Information System (LLIS), because it was difficult to navigate and it failed to provide them with useful lessons.

Improving the architecture of a database is simple enough. The real problem was it did not provide useful information. If the system provides value, it's likely to get used regardless of deficiencies in its navigation.

Unfortunately, one inference suggested by the GAO report was that NASA managers don't want to share knowledge. Frankly, the author finds that at odds with what he has seen since he started working with NASA project managers on ASK. Who doesn't recall during the Mars encounters the images inside the control room at the Jet Propulsion Laboratory of people congratulating one another, jumping up and down even, when the Spirit and Opportunity rovers delivered signals of their successful landings--or for that matter, the images of agency members consoling one another as they mourned the loss of the Space Shuttle Columbia crew? Clearly, this is not an organization devoid of camaraderie and shared mission.

The success of ASK--6000 print issues published bimonthly, another 8,000 people receiving the electronic edition--should not surprise anyone who understands that a lessons learned system that is genuinely useful is a sure winner in any organization. In the case of NASA, they know a good thing when they see it.



The Challenge to Your Organization

There is one thing we should recognize about project work in most organizations: an overwhelming number of people who perform this work are practitioners. Across all levels of project work, people learn quicker, smarter, and far better by reflecting on their workplace experiences than by consulting theory. Either they can do this in the privacy of their own thoughts, or far more effectively, as this author would argue, by reflecting on their experiences with--and listening to the experiences of--their colleagues.

Again, stories stimulate reflection. "How might I address that issue?" asks the practitioner of him or herself upon hearing a colleague tell a story about a workplace challenge. That, in turn, leads to the obvious next step: "How have I handled similar challenges?" This holds true not only at NASA. The author is confident that you have witnessed this, too, in your own organization among all levels of practitioners.

"Because professionalism is still mainly identified with technical expertise," writes Donald Schon in The Reflective Practitioner, "reflection-in-action is not generally accepted--even by those who do it--as a legitimate form of professional knowing" (Reference 6). We should appreciate Schon's insight here, because it may help to explain why it has been so difficult for other storytelling initiatives to get off the ground, even where there are enthusiastic proponents for storytelling within the organization.

There is much to be said for consistency. Our work on ASK Magazine did not come crashing out of the gate with broad acceptance throughout NASA. It was new, it was different, and there was already some cynicism, based on the ineffectiveness of the LLIS, about initiatives aimed at providing lessons to help practitioners to do their jobs better. That we arrive in NASA mailboxes every two months with stories by the "best of the best practitioners" has gone a long way towards winning over skeptics.

Like most fledgling initiatives, we began on a small scale, starting initially as a web-based publication and with a distribution list of a few hundred, mostly NASA project managers who were already happy customers of other APPL products. A print publication followed for marketing purposes, and to address the most common observation about ASK's early issues: "I wish there was a way I could read these stories while I was on the plane."

ASK was fortunate to have a sponsor in APPL Director Dr. Edward Hoffman, whose own work on storytelling with ASK Editor-in-Chief Dr. Alexander Laufer (including a book, Project Management Success Stories) (Reference 7), gave him faith and confidence that with time the magazine would find wide acceptance, and it has. Testimonials about the efficacy of ASK lessons run the gamut from cog engineers to center directors and associate administrators. The push is on throughout the Federal government and across industry to capture knowledge, and find mechanisms like ASK to get that knowledge to the people who need it. And so that's our story. Are you reflecting on yours?



References

1. ASK Magazine can be accessed at http://appl.nasa.gov/ask


2. Schon, D.A., The Reflective Practitioner, Basic Books, New York, 1983; Schon, D.A., Educating the Reflective Practitioner, John Wiley & Sons, Inc., San Francisco, 1987.


3. D.A. Schon, (1983), p. 62.


4. D.A. Schon, (1987), p. 17.


5. http://www.govexec.com/dailyfed/0202/021102m1.htm


6. D.A. Schon, (1983), p. 69.


7. A. Laufer and E. Hoffman, (2000) Project Management Success Stories: Lessons of Project Leaders, John Wiley & Sons, New York.