Comments Offered To The Agency On Airman Certification Standards

(Editor's note: The following are Rod Machado's comments added to Docket FAA-2013-0316. They are posted here with the author's permission)

The desire to reduce general aviation accidents is a noble and worthwhile impulse and one about which there can certainly be no disagreement. How we achieve this aim, however, is a matter for discussion. In my opinion, the attempt to accomplish this objective through a rewrite and modification of the Practical Test Standards (PTS) misses the mark by a wide margin.

I believe the proposed Airmen Certification Standards (ACS) will complicate and increase the cost of flight training while having little if any effect on general aviation’s safety record. The most likely result of the proposed change is fewer private pilots, further decline of the general aviation industry infrastructure, and no positive impact on flight safety.

Let’s begin at the beginning with the stated goal of the proposed ACS concept.

The Goal That Wasn’t

In the introduction to the private pilot ACS, you write:
“The goal of the airman certification process is to ensure the applicant is ready to safely manage the risks of flight as pilot-in-command, consistent with the privileges of the certificate or rating exercised.”

When making such significant changes to the testing process as are proposed here, these changes should be directed at achieving a clearly-defined goal. Such a goal is one with quantifiable and measurable metrics by which to evaluate progress toward achieving that goal as well as the effectiveness of the strategy used to do so. It seems to me, however, that a goal to, “...safely manage the risks of flight” is abstract and difficult to quantify, much less measure.

Had you stated this goal in terms of a numerical change in the accident rate you would have been on solid footing in terms of the goal’s clarity and accountability. Unfortunately, the goal as presented fails to provide a clear definition of objectives.

Here is a goal that I consider both definable and measurable. The goal of the new airman certification process is to reduce the combined fatal and non-fatal GA (aircraft <6,000 pounds) accident rate of January 1, 2012 by 50% no later than 2020. Regardless of whether or not this is a reasonable goal (unfortunately, I don’t believe it is), it does meet the requirement for clarity and measurability. Unlike the present ACS goal, this one provides a clear definition of objectives.”

If we don’t know precisely what we want to accomplish, how will we form a practical goal-achieving strategy, and how will we know if the means we employ toward achieving that ambition are effective? The answer is, we won’t. All we have here is an intention to do something well intentioned.

Let’s Pretend that Risk Assessment Isn’t Knowledge

One of the main features of the proposed ACS is the separation of risk assessment into its own special category that’s distinct from the categories of skills and knowledge found in the present PTS. The assumption is that the knowledge of risk assessment deserves special treatment during the practical testing process. I believe this to be a redundancy that will produce little or no positive results.

The demonstration of a practical skill is a concomitant demonstration of risk management as it relates to that skill. For instance, if a private pilot applicant can’t handle a simple crosswind, then it should be clear that he or she is not capable of assessing or managing risk in this area. On the other hand, if this applicant can competently manage crosswind takeoffs and landings, why should we not assume that he or she also has sufficient knowledge of risk assessment skills in the area of crosswinds?

Since we’re giving the applicant a practical test, it’s entirely reasonable to make this assumption, mainly because there’s a practical limit to the amount of time that can be spent testing the applicant. Designated pilot examiners (DPEs) make similar assumptions about a pilot applicant’s knowledge on every checkride. Let’s remember that the practical test is not a complete and comprehensive assessment of a pilot applicant’s total knowledge and skill. A comprehensive test of that nature would most likely require many days of observation, which is impractical. DPEs have no choice but to make generalized assumptions about an applicant’s overall competence from limited observations of that applicant.

Where did we come up with the idea that risk assessment isn’t knowledge and therefore needs to be isolated as its own category in the proposed ACS? This is a very strange idea. Risk assessment is knowledge. The risk assessment associated with a physical skill can be evaluated during the performance of that skill. That’s the purpose of a practical test. Risk assessment associated with aviation concepts (other than physical skills) can be assessed during the knowledge exam.

In fact, this is exactly what you state in the proposed Private Pilot ACS on Page 3-1. You write:
“The FAA does not expect the evaluator to test every knowledge and Risk Management element of the Practical Test, as the Knowledge Test assessed the applicant’s mastery in these areas.”

Why is it not reasonable to assume that the knowledge exam previously taken by the applicant has already evaluated his or her risk assessment acumen? Why do you need to test the applicant’s risk assessment knowledge again on the practical test when the applicant is expected to provide a physical demonstration of that knowledge? This seems to be a waste of time. It’s also something the ACS proposal suggests is unnecessary. If you feel that pilot applicants need more testing in the risk assessment area, then present an argument in support of increasing the questions on the knowledge exam. It seems unreasonable to separate risk assessment into its own category without a practical reason for doing so.

The Business of Do-Something Activities

It seems to me that when the FAA or the aviation industry become frustrated at their inability to reduce aviation accidents, they engage in what I call “do-something activities.” This is where we do something (or anything) in an attempt to reduce the aviation accident rate. These activities usually involve untested, untried, feel-good notions that seem to have little or no positive effect on aviation accidents but do seem to do harm on occasion, as I’ll explain shortly. These activities are well intentioned, but I’ve long realized that having good intentions is a highly overrated virtue.

For instance, when Aviation Decision Making (ADM) from Transport Canada first appeared in the early 1980s, it was a complete package. It contained many reasonable and practical principles (hazardous thought patterns and antidotes, for example) along with selected scenario training activities and useful risk assessment knowledge. It was highly practical for instructors to use, and it wasn’t a burden on the flight training process. In short, it did more good than harm.

Unfortunately, over the years, individual ADM concepts have been isolated, extracted then supersized into programs that were individually promoted as cure-alls for the relatively constant aviation fatality rate.

So how are those programs working for us so far?

After 10 years of FAA-Industry Training Standards or FITS (an example of one of these programs), there has been no discernible effect on general aviation safety. In fact, the fatal accident rate actually ticked up slightly during those 10 years. When you ask the retired FAA person involved in the creation and promotion of FITS about the lack of proof that these concepts actually work, he suggests that instructors are either not using the principles of FITS or not applying these concepts correctly. Really? Isn’t this a similar rationale used by the faith healer when he fails to heal? He says, “Well, the recipient of my touch just wasn’t ready to be healed.” This is a peculiar response to the failure of such a well promoted program.

The fact is that isolating risk assessment as its own special distinct category that will be applied to each and every required Task in the proposed ACS, seems to be a do-something activity of sorts. This activity appears to bloat and obfuscate the practical testing process. If it produces any discernible effect on aviation safety, it will likely be to increase the difficulty and expense for someone to obtain a private pilot certificate, thus resulting in producing fewer pilots.

Now, I’m not one to criticize without offering an alternative plan.

If you really want to reduce accidents then follow the recommendations in the 2005/2007 Human Factor Analysis and Classification System (HFACS) study. The HFACS study says:
“The study demonstrated the that greatest gains in reducing aviation accidents could be achieved by reducing skill-based errors.” The study also indicates that, “...the data indicated that skill-based errors were associated with the largest portion of GA accidents (79.2% of the 14,436 GA accidents)....”

The HFACS study suggests that we are no longer teaching pilots how to physically fly airplanes. Instead, we’re teaching them how to mimic the behavior of airline pilots in hopes of achieving the low accident rate of commercial aviation. The problem is that nothing about flying big airplanes pertains to flying small ones, while everything about flying small airplanes pertains to flying big ones. Were we to emphasize stick and rudder skills during the training and the practical testing process, we would most likely see immediate gains in aviation safety. It’s clear to me that the proposed ACS is a step away from testing an applicant’s stick and rudder skills, not a move towards it.

We obviously need to teach aviation decision making (which includes risk assessment) and test an applicant’s knowledge in these areas. Clearly this knowledge can have a direct effect on the fatal accident rate. So teach ADM as knowledge (as the newer FAA handbooks do quite well) and test for comprehension and understanding in the knowledge exam. The current FAA handbooks are quite capable of providing knowledge training in this area.

Finally, let’s consider the consequences of moving from the PTS to the proposed ACS.

• The Cost and Disruption to Flight Training
• The new ACS proposal appears to significantly increase the complexity of flight training in several areas. Consider that the current private pilot PTS for both single-engine land and seaplane consists of 45 pages. The proposed private pilot ACS contains 82 pages and only addresses the single-engine land certificate.

The proposal promises to expand the document way beyond 82 pages with the inclusion of knowledge exam questions (although the proposal isn’t at all clear how this will occur). If that weren’t enough, the ACS proposal says, “The ACS uses Notes to emphasize special considerations.” I can only imagine how this document will grow with time as individuals make modifications to the ACS in an attempt to influence pilot development.

What is clear is that the proposed ACS is nearly twice as long as the PTS. In fact, the proposed private pilot ACS looks more like a bloated syllabus rather than a testing standard. In an attempt to create a document that will, “...ensure the applicant is ready to safely manage the risks of flight” the working committee has followed the natural trajectory of this ambition and (apparently) produced a syllabus, instead. The PTS is not a syllabus and was never intended to be one.

In the introduction to the proposed private pilot ACS, you write:
“The FAA encourages applicants and instructors to use the ACS to measure progress during training, and as a reference to ensure the applicant is adequately prepared for the knowledge and practical tests.”

Isn’t the purpose of a syllabus to ensure that the applicant is adequately prepared for the practical test? Isn’t the purpose of a practical test standard to provide the applicant with the behavioral standards used for evaluation during the test? It seems to me that the proposed ACS has morphed from a practical testing standard to a practical training standard.

I’m concerned that the FAA has failed to consider the consequences of the ACS proposal. If flight instructors have to adhere to the additional requirements of the proposed ACS, it will certainly lengthen the process of flight training. How can it not do so? After all, the proposed private pilot ACS is longer than most private pilot syllabi. It’s certainly not unreasonable to suggest that a traditional private pilot applicant will now find his or her flight training times increased, thus increasing the cost of earning a private pilot certificate. The time it takes a DPE to conduct a checkride is likely to increase, too. This will result in further increased cost and time for the practical test.

Where’s the Proof?

What’s missing here is proof that altering the PTS will accomplish the objective of making aviation safer. What’s present, however, is a great deal of abstract language used to support these ideas. The moment I encounter this type of language my suspicion detector goes off immediately.

For instance, in the “Building an ARC” introduction, you state definitively:
“...this approach would improve and integrate testing and training by clearly mapping aeronautical knowledge and risk management to the flight proficiency skills as defined in the PTS.” This leaves me with the following unanswered questions:

• How will this approach improve testing?
• Improve testing compared to what?
• Why does our present testing process need improving? Give me a good reason.
• How does this approach integrate testing?
• Why is it that we need to integrate testing?
• What proof do we have that integrated testing is better?
• What do you mean by mapping knowledge to skills?
• What will mapping knowledge to skills accomplish?

I mean no disrespect here, but as I read the ACS proposal, you haven’t answered any of these questions. You’ve just made bold claims that this process will work because it’s being done in accordance with the FAA’s Safety Management System Framework (SMS). Well, there goes my suspicion detector again. How does SMS methodology provide “...a systematic approach to achieving acceptable levels of safety risk?” You’ve not clearly defined your goals in the first place, then you’ve supported abstract ambitions with abstract language. Sorry, but you’ve given me no confidence that the keepers of this proposal actually know what they want to accomplish.

Perhaps the most important question you haven’t answered is, “How much will this cost in terms of money and manpower?” It’s not as if the FAA is flush with funds these days. The logical consequence here is that you’ll take many of the individuals who are already doing a fantastic job (in my opinion) and task them with working on the ACS. That means the good works they are presently accomplishing will likely suffer. I’m not at all convinced that you have considered the consequences of this proposal.

What you’re doing is taking the PTS—a document that has been refined and improved over the years to its present level of sophistication—and suggesting changes while offering no evidence that aviation will be made safer as a result. Common sense, however, suggests that it will make flight training more difficult and less affordable. This would not be a good outcome.

Finally...

I’ve been in the business of preparing pilots for both the knowledge and practical exam for nearly 40 years and have some familiarity with the workings of the FAA in this area. As I see it, the PTS and the FAA’s present bank of knowledge questions do exactly what they’re supposed to do. Yes, I’m sure there are a few questions that not valid or reliable on the knowledge exam, but this is the exception and not the rule. At least that’s what student test-takers are telling me. I also happen to think the world of the individuals at the Airmen Testing Standards branch of the FAA. These folks do fantastic work, in my opinion.

My recommendations:

1. Leave the PTS exactly the way it is. These are practical, useful and easily understood documents that have undergone positive development for many years. They shouldn’t be replaced by the ACS unless there’s solid evidence suggesting an increase in aviation safety will be the result. If you want to change to the ACS concept, then do so based on proof, not public demand.

2. If there are individuals or groups who are concerned about the validity and reliability of knowledge exam question, then provide them with a means of examining the knowledge test areas in question at a local testing center. This should be relatively easy to accomplish. What shouldn’t be allowed, however, is for these individuals to have permanent or long term access to FAA knowledge test questions. Nothing good can come from this. I was teaching ground school in the late 1970s when the FAA first released the FAA testing questions to the public as an Advisory Circular. In my opinion, this did far more to hurt pilots than it did to help them. I certainly wouldn’t have wanted my doctor to have had access to the exact questions he or she would experience when sitting for the medical licensing exam. Would you?

3. Since we have written standards for the practical flight test, why shouldn’t we have delineated standards for the knowledge exam? I’m speaking of having a clear and comprehensive idea of the required areas of study for the FAA knowledge exam. Yes, we currently have a knowledge test guide, but my impression of it is that it lacks sufficient specificity to be practical. For instance, the present knowledge test guide suggests that a private pilot applicant needs to understand the concept of “moisture” as presented in AC-006 (Aviation Weather). Unfortunately, the word moisture appears 48 times in that book and occurs in every chapter, including the ones titled, “High Altitude Weather” and “Arctic Weather” Do we really expect a private pilot applicant to study these chapters in preparation for the private pilot knowledge exam? If we can define learning objectives in the PTS in behavioral terms, why can’t we define the required study in similar, precise, and practical terms? Perhaps the creation of Knowledge Test Standards (KTS) for different certificates/ratings that clearly identify the topics students must study. For instance, the concept of runway hot spots is something with which students should be familiar. How is a student supposed to know that these areas are now (or will be) covered on the knowledge exam?

4. If you feel there’s a need to test applicants on risk assessment knowledge, then make the case that the knowledge exam should contain more questions. It’s my guess that the use of 60 questions (the current number of questions on the private pilot exam) was based on early 20th century criteria that are no longer valid. So provide proof or at least a strong argument for this change. I think you’d be surprised at how many people might actually support this type of change if you can present proof that it will make aviation safer.

Sincerely and with respect,
Rod Machado
June 26th, 2013

FMI: www.rodmachado.com/