Adam Caudill

Security Engineer, Researcher, & Developer

Crew Resource Management for Security Teams

Over the last year or so, I’ve become quite a fan of Air Disasters, a television show dedicated to analyzing plane crashes and similar incidents. As I watched the show, I started seeing many ways that the lessons and procedures around aircraft safety also apply to running a security team; this valuable and hard-won wisdom, often born out of tragedy, can be of significant impact if appropriately applied. In this article, I will explore Crew Resource Management and how it can be applied to Information Security to make teams run better. Hopefully, these insights help you achieve more and fulfill the critical missions we are entrusted with.

What is Crew Resource Management? #

In 1977, the deadliest airline disaster in history occurred at Tenerife, when two 747s (KLM 4805 and Pan Am 1736) collied and cost 583 people their lives; this happened for a variety of reasons, though would have been prevented if the captain had listened to his team. The following year United flight 173 crashed on the way to Portland, Oregon; the flight crashed because the captain was distracted by a malfunctioning landing gear and ran out of fuel (despite warnings from the crew). These accidents are the direct result of human errors, which account for as many as 80% of all accidents. These events, along with many others, led to the development and implementation of Crew Resource Management (CRM) and forever changed the way airliners are operated.

The implementation of CRM has drastically reduced the number of accidents that occur and has made air travel vastly safer than decades ago.

The value of these lessons has been applied in a number of other areas, such as firefighting, ship operations, air traffic control, healthcare, and others. To not take advantage of these lessons would be a missed opportunity for the security community to advance our practices.

What, exactly, is CRM? #

CRM is intended to address a number of human factors, which is to say, preventing or mitigating entire classes of human errors through training focused on group dynamics, leadership, interpersonal communication, and decision-making. The focus is on seven specific areas:

  • Mission Analysis
  • Situational Awareness
  • Communication
  • Adaptability / Flexibility
  • Decision Making
  • Assertiveness
  • Leadership

These will be discussed in more detail below.

It should be noted that CRM alone is not a silver bullet; CRM compliments technical skills — to quote an FAA document on the topic:

Demonstrated mastery of CRM concepts cannot overcome a lack of proficiency. Similarly, high technical proficiency cannot guarantee safe operations in the absence of effective crew coordination.

CRM Focus Areas #

As CRM focuses on several different areas, we’ll look at each individually and discuss how they apply to our security work and the environments we work in.

Mission Analysis #

A leader is responsible for understanding the task at hand, and all its potential risks and benefits, and developing plans to safely meet the objectives.

When setting out on any task, it’s critical to fully understand not only the work, but what could go wrong. It’s disturbingly easy to create new issues in the process of trying to fix something; not understanding the implications of a change is responsible for a substantial number of security issues.

Situational Awareness #

Loss of awareness of what’s happening around you can easily end in tragedy. Situational awareness must be a constant state of understanding what’s happening, combined with the alertness to notice changes.

Understanding what’s going on can be quite a challenge, especially in a complex and fast-moving environment — but when we lose sight of what’s going on, there’s a substantial risk of things going wrong. There are, thankfully, some warning signs that we can watch for to identify when we risk losing our situational awareness.

Ambiguity #

Do we understand the details clearly? Do we have enough information to make a sound decision? Can we trust the information we have? Can it be interpreted in more than one way?

All of these are warning signs that we aren’t working with sufficient information, and caution is required. While we can’t be paralyzed by bad data, we must evaluate what we know carefully and move ahead with care.

Distraction / Overload #

Can you focus on the task at hand, or are you constantly being pulled in multiple directions at once? Can you complete your work correctly, or are you constantly leaving things half-done? Do you know where you left off from your last task?

If we can’t focus and complete routine tasks because of so many other things happening, it becomes impossible for us to understand what’s going on and where things stand. While we have to remain flexible to changing priorities, we can’t push ourselves so far that we lose sight of where we are.

Fixation #

On the opposite side of distraction is fixation, becoming so focused on a single task that we can no longer see the big picture — missing important information and events because we become blind to everything else.

This is particularly dangerous as it makes it easy to be confident that things are under control, when in reality, we have no idea what’s actually happening.

Complacency #

If we become too comfortable that everything is going well, we can let our guard down and miss the signs that something is going wrong. It’s vital that we are always alert and watching for signs of trouble — constantly aware of what’s going on, and the risks that these events present.

Unresolved Discrepancy #

When something is odd, do you follow it down to the root cause, or ignore it and focus on other tasks? If we ignore things that aren’t right, we may miss important information about what’s going on — these discrepancies are essential clues to help us establish and maintain awareness. We can’t ignore things when they don’t add up; there’s always a reason for these situations, and we need to understand what it is.

Lack of Control #

In aircraft terms, it would be “nobody is flying the plane” — in our world, we would say that nobody is actually in charge. This can be anything from abandoned tickets to a lack of leadership, a failure to ensure that the work that needs to be done is actually being done. Thanks to huge backlogs and understaffed teams, it’s easy for security to become disconnected from the rest of the company. Projects and tasks move forward without anyone from security involved — and this is a setup for failure.

While I don’t suggest that security should be in control of everything, but I do suggest that without security being involved, efforts can go in any direction because nobody is guiding the security properties or handling the threats that arise.

Communication #

One of the most vital skills, if not the most critical skill, in working in a team environment is effective communication. In reality, we are all communicating constantly, but too often, communication isn’t actually effective, and as a result, things can quickly go wrong because the information isn’t received as intended. There can be a wide variety of reasons, on either side, that causes these simple breakdowns in understanding — but the result can be incredibly damaging as it increases the likelihood of an error and makes it impossible for us to act effectively.

One method that CRM teaches is to use a 5-step process to make an assertive statement:

  • Opening - A clear statement to indicate who the message is directed towards to ensure that you have their attention.
  • State the Concern - Clearly articulate what it is that you think is wrong, what you’ve seen, or what you’re worried about.
  • State the Problem - Now that you’ve made it clear what you’re worried about, explain the issue as you understand it. This helps to ensure that both parties are on the same page.
  • Solution - Offer a solution to address the problem. Make it clear that action is needed and that you are suggesting a specific course of action to address the issue.
  • Agreement - Clearly ask for approval or consent to implement the suggested solution.

All of these components play an essential role in communicating an issue and determining how to respond to it; if any element is left out, there may be confusion about what’s going on, why something would be done, or if the solution should be implemented. This technique also opens the door to further discussion around other solutions, as it ensures that everyone has the same understanding of what’s going on and what options may exist.

It’s important to understand that many factors play into effective communication. Developing this skill may require changes to personal habits, as well as changes to culture to allow this type of communication to exist and be effective.

Communication Failure Types #

There are many ways that communication attempts can go wrong, and thankfully for us, some of these have been documented to help us avoid them ourselves and spot them when they occur. We’ll focus on both errors from the person providing the information, and receiving the information.

Provider #

There are a variety of ways that we can fail to effectively communicate information when we are trying to provide it to another, such as:

  • No frame of reference - If you provide information without context or to a person who doesn’t possess the knowledge you assume, they may not understand the information or processes it as intended.
  • Missing information - If there is information or ambiguity, it’s easy for the receiver to come away with a different or incorrect understanding.
  • Inserting bias or opinion - If you include your personal opinion or show signs of bias, the information may seem less trustworthy, and the facts may not be accepted as such. It’s essential to keep personal views out of critical communication or clarify what’s a fact and what’s opinion.
  • Body language & tone - When speaking face to face, it’s important to be mindful of our body language and tone, as we may unintentionally alter the message or how it’s perceived.
  • Unwillingness to repeat - In high-stress situations, there is often a desire to move quickly and not waste time, and repeating something can undoubtedly feel like wasting time. However, we must be willing to repeat, in full, as needed to ensure that everyone understands what we are trying to convey. Many errors are introduced by repeating information and leaving bits out during that process.
  • Disrespectful or hostile communication - If the message is perceived (regardless of intent) in a disrespectful or aggressive manner, it’s unlikely that effective communication will occur. While it’s easy to become upset in times of high stress, this can lead to compounding mistakes and make a situation harder to recover from.
Receiver #

Just as there are many errors that the provider can make, there is also a number that the receiver can make:

  • Prejudice - If we make up our mind before understanding all of the information, it will impact and change our understanding and lead to new mistakes. It’s essential to have as complete a picture of the situation as possible before proceeding.
  • Attention - It’s far too easy to become distracted or not give a person full attention. This can lead to missing important parts of a message and either not understanding the situation or having a flawed understanding. Listening requires concentration and focus, and this fact must not be ignored.
  • Thinking ahead - In our field, it’s easy to hear someone and start to extrapolate or make assumptions about where they are going — if these assumptions are incorrect, it can lead to adverse outcomes. If you hear someone say “let me finish” or “hear me out,” it’s a clear warning that this is occurring.
  • Ignoring non-verbal cues - People communicate in a variety of ways, including voice tone and body language. These additional methods may contain valuable insight or other information that isn’t being presented verbally; it’s important to pay attention to these and adjust our understanding accordingly.
  • Clarification - Failing to ask for clarification, failing to seek more details, failing to gain a useful understanding all lead to making a situation more complicated. There’s no shame in asking someone to explain; while we may feel like we should already know something, missing the chance to gain understanding and moving ahead with a flawed perspective makes things worse.
  • Disrespectful or hostile communication - Knee-jerk reactions are often incorrect and lead to further mistakes. Becoming angry or seeking to assign blame often lead to additional errors. Information should be received and carefully considered, and thoughtful responses given.

Managing Tone #

When communicating with others, especially during a high-pressure event, it’s important to be aware of the tone we use and avoid charged language. If we accuse someone of making a mistake, or making a suggestion that isn’t wise, we put that person on the defensive, and communication becomes ineffective.

The focus must be on addressing the situation, not on placing blame on anyone. Making this mistake can easily take a situation and make it far worse.

It doesn’t matter who’s to blame; what matters is how we respond.

Adaptability / Flexibility #

Adaptability/Flexibility refers to the ability to alter one’s course of action contingent upon or as a function of another’s action and/or situational demands.

In most cases, it’s best to have clear procedures that are followed; this allows everyone to understand their role and know exactly what’s expected of them — that is, until procedures aren’t enough. No matter how well we plan, what procedures and policies we enact, there will always be situations where we need to adapt. There will always be situations where we need to adjust rapidly, alter plans or behavior, find ways to help others, and find constructive options under pressure.

Decision Making #

Effective decision-making refers to the ability to use logical and sound judgment based on the information available.

Making sound decisions is quite obviously critical to our work, and there are a few guidelines that CRM offers us to make better decisions:

  • Accessing the problem.
  • Verifying information.
  • Identifying solutions.
  • Anticipating consequences of decisions.
  • Telling others of the decision and rationale.
  • Evaluating the decision.

These should be fairly familiar as they also factor into the guidelines for effective communication that we covered above; good decision-making and effective communication are very closely linked. Without effective communication, good decision-making isn’t possible.

Imperfect Information #

While the best decisions occur when everyone is fully informed and has a clear understanding of the situation, the reality is that these situations are rare. Most decisions are made with information missing. While sometimes derided, there is an apt quote that explains this well:

… there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns—the ones we don’t know we don’t know. - Donald Rumsfeld

There are things we know and understand, there are things we don’t know or don’t understand (but we recognize they exist), and things that we don’t know and have no idea that we don’t know. When evaluating information, it’s important to keep this in mind, as this concept plays a vital role in determining what is and what isn’t actionable.

Assertiveness #

Assertiveness refers to the willingness to actively participate, and the ability to state and maintain your position, until convinced by the facts (not the authority or personality of another) that your position is wrong.

This is a particular challenge in our field due to the variety of personality types that exist, and how many people are excessively assertive (often without realizing) or are too introverted to be comfortable being assertive. While this is a challenge, it’s one that must be addressed.

CRM gives us a few ways that team members can assert themselves:

  • Providing relevant information without being asked.
  • Making suggestions.
  • Asking questions as necessary.
  • Confronting ambiguities.
  • Maintaining their position when challenged.
  • Stating opinions on decisions/procedures.
  • Refusing an unreasonable request.

Of course, it’s important to remember that, as discussed above, all communication needs to be respectful, and being assertive does not mean that the chain of command can be ignored. Team members should be empowered to do the right thing, though leaders are ultimately responsible for everything a team does.

Most conservative response rule #

Occasionally there is a disagreement in the cockpit that cannot be resolved due to lack of information. It is best to agree in advance to take the most conservative action in these situations until additional information is available.

What happens when there are multiple options (with different outcomes), and we aren’t sure which path to take? Following a rule like this helps to provide guidance and settle disagreements so that progress can be made quickly.

Two challenge rule #

In extreme situations, if the pilot does not respond to two demands (e.g., “Waveoff, waveoff!”),the copilot should take the controls.

Does everyone on the team know what to do when they don’t get a response? When there’s no reaction (leaders are unavailable, for example), is there a way to handle issues, or does everything freeze until a leader is finally available?

In fast-moving situations, it’s crucial that everyone knows what to do and is able to act, even when others don’t.

Sandbag syndrome #

The Sandbag Syndrome is based on a comforting premise that one or more other crew members have the situation under control and are looking out for your best interest. It can be experienced by any crew position, resulting in that person being “along for the ride.”

This is likely the most critical part of assertiveness, as it’s incredibly common and creates many problems. Assuming that someone else will handle something works fairly often, because someone else may well handle it — but there’s no way to be confident; it could be missed completely. Each and every team member should assume ownership for issues that they see, even if it’s just confirming that someone else is actually aware.

This is not to say that people that fall into this trap are lazy or avoiding work, but they may be uncomfortable jumping in, they may not be sure who is responsible (if anyone), or maybe they are worried about what happens if something goes wrong. There are a hundred reasons that someone will sit back and expect someone else to do something, but it’s a problem waiting to happen.

If there is a single thing that should be achieved encouraging assertiveness, it’s addressing this specific problem.

Leadership #

Leadership is the ability to direct and coordinate the activities of other crew members and to stimulate the crew to work together as a team.
Being a good leader involves inspiring your crew to work up to their potential; a good leader can bring out the best in their crew. The leader is in control of the situation and has certain responsibilities.

Let’s start by looking at what CRM tells us a leader should be able to do:

  • Direct and coordinate the crew’s activities.
  • Delegate tasks.
  • Make sure the crew understands what is expected of them.
  • Focus attention on the crucial aspects of the situation.
  • Keep crew members informed of the mission information.
  • Ask crew members for mission-relevant information.
  • Provide feedback to the crew on their performance.
  • Create and maintain a professional atmosphere.

None of these should come as a surprise; this is a brief but accurate portrayal of what leadership is. Though there’s much more to being a good leader than a few bullet points, a topic I’ve written about before.

Authority #

In CRM, authority is reinforced through a few practices:

  • Ensuring mission effectiveness.
  • Fostering respectful communication.
  • Establishing clearly defined goals.
  • Including crew input.
  • Establishing clear assignments.

As a leader, it’s essential to take responsibility and ownership while also listening to your team and taking full advantage of their knowledge and perspective. Establishing and maintaining authority does not mean excluding others, but acknowledging responsibility.

Mentoring #

Every leader is responsible for mentoring those on their team, sharing knowledge and experience they’ve gained to help their team members grow. This helps to ensure that the team not only maintains technical competence but helps to establish trust in leaders.

Conflict Resolution #

Conflicts can and (and do) arise in every team, and can lead to growth if appropriately handled. It’s up to the team leader to resolve conflicts and act as a mediator when needed. It’s important to listen attentively, don’t become emotionally involved, and gain a clear understanding of the underlying cause of the conflict.

Followership #

In an organization, essentially everyone is a follower, even if they are a leader as well. As such, it’s essential to understand how to follow effectively in addition to leading effectively. Being an effective follower isn’t passive, but a conscious effort and one that requires one accept certain responsibilities, such as:

  • Respecting authority.
  • Keeping ego in check.
  • Properly balancing respect and assertiveness.
  • Requesting clear assignments.
  • Reporting on the status of tasks honestly and clearly disclosing any issues.
  • Acknowledging mistakes.

Under the CRM model, followers are empowered to challenge leaders when a situation demands it, but this is dependent on followers exercising this right responsibly.

Hazardous Attitudes #

Keeping an eye out for hazardous attitudes — attitudes that can lead to mistakes — is helpful to spot issues before they happen. Some examples are:

  • Anti-authority: “Don’t tell me.”
  • Impulsivity: “Do it quickly.”
  • Invulnerability: “It won’t happen to me.”
  • Machismo: “I can do it.”
  • Resignation: “What’s the use?”
  • Pressing: “Let’s hurry up so we can go home.”
  • Airshow syndrome: “I’m going to look amazing!”

Personal Factors #

It’s vital as a leader to understand that you are working with humans, not machines. Many factors heavily impact how well a person works, such as:

When working with your team, it’s important to remember that these factors may exist and may be impacting them in ways that aren’t immediately obvious.

Conclusion #

I do hope that this has been useful and thought-provoking. While not everything in CRM applies quite the same way to the situations we face, hopefully, you can find ways to use these lessons in your own work. There are a vast number of resources on CRM, and I encourage you to look into it more.

Adam Caudill


Related Posts

  • Leading Experts

    A friend of mine recently asked for my thoughts on leading people who have more experience or expertise in a topic than they do; this is an important question and one that I felt deserved more thought and exploration. Leading people can be difficult, but when leading people that know more than you do about a given topic, it’s a different challenge. This was particularly well-timed, as I’ve found myself in just that situation, as I’ve just hired a specialist in incident response.

  • Best Practices vs Inane Practices

    or A Full Vindication of the Measures of Security Practitioners, from the Calumnies of their Enemies; In Answer to A Letter, Under the Signature of A. Gwinn. Whereby His Sophistry is exposed, his Cavils confuted, his Artifices detected, and his Wit ridiculed; in a General Address To the public, And A Particular Address To the dedicated members of the security community. Veritas magna est & prœvalebit. Friends and Colleagues, It was hardly to be expected that any man could be so presumptuous as to openly controvert the equity, wisdom, and authority of the measures, adopted by the practitioners of information security: a group truly dedicated to the protection of business and individuals around the world!

  • When Hashing isn’t Hashing

    Anyone working in application security has found themselves saying something like this a thousand times: “always hash passwords with a secure password hashing function.” I’ve said this phrase at nearly all of the developer events I’ve spoken at, it’s become a mantra of sorts for many of us that try to improve the security of applications. We tell developers to hash passwords, then we have to qualify it to explain that it isn’t normal hashing.

  • I Love My Job

    I love what I do, and I work with a great team. While it’s still far from perfect; I can say that I do love my job. For the last couple weeks though, I’ve had to remind myself of this several times. I’m sure we’ve all done it, in this industry it’s hard to avoid. You read an email or receive a phone call and repeat the mantra “I love my job, I love my job, I love my job.

  • Declaring War on Ransomware

    It’s time for everyone from the industry, developers, and the government to declare war on ransomware and make it as hard as possible for them to ply their insidious trade. There have been false starts and baby steps, diligent fighters without enough resources, and vendors that have only given a nod to the issue. It’s time to use every tool reasonably available to stop this scourge. For so many in the industry that have dedicated so much of their time and effort to this fight, this statement may seem to diminish their efforts, but that is not my intent.