Address Concerns About Knowledge Tests

The National Association of Flight Instructors (NAFI) joined FAA staff, other industry representatives, and university experts in Oklahoma City recently to review how the agency creates airman knowledge-test question banks-a process that corrected problems in the recently revised Fundamentals of Instruction test.

Education experts from university aviation education programs, including Embry-Riddle Aeronautical University, Liberty University,  the Professional Aviation Board of Certification, University of North Dakota, the University of Oklahoma, and Western Michigan University, participated in the meeting at the request of NAFI and AOPA, as a follow-up to discussions held several weeks ago. The group specifically addressed the Fundamentals of Instruction test; rates of failure for that test have increased significantly since the bank of questions used to create the test was updated on February 14, 2011, Blair said.

"We took a closer look at the specific questions that were added to the test bank and evaluated the test question structure, answer options, and reference for each question," said NAFI Executive Director Jason Blair. "Six questions were identified for removal from the test bank, and a few others that were identified for revision. The FAA immediately addressed those questions, and it will review the results of applicants who failed the test on the basis of these questions."

Those applicants will be contacted directly by an FAA staff member, and a revised knowledge test score will be issued, Blair said.

In spite of that correction, the group noted that many of the new questions are directly drawn from the most recent update of the Aviation Instructors Handbook. Some of these questions continue to result in poor application performance, but they're well written and based solidly on the content in the handbook, Blair said. That reinforces that it should be used as the primary text for applicants preparing for this particular knowledge test. Review of results from other knowledge tests suggest that both pass rates and overall scores haven't changed enough to warrant further review of other test banks, Blair said.

"Both the FAA staff and industry representatives strongly support an approach where students study for airman knowledge tests by developing their knowledge of the material, not by studying specific test questions," Blair said. "We're working together to create an understanding of how to best prepare applicants for tests while improving overall knowledge and safety in our aviation system."

In addition to the test-question review, university experts also addressed test question development and validation, and they provided feedback on some methods of test development that are generally accepted in the educational community. The FAA recognizes that it can learn from the education community, Blair said. As a result, it will continue this discussion with an industry steering committee it expects to form in the near future, which will be comprised of educational and industry representatives who can provide feedback on test development and validation.

"NAFI strongly supports the continued effort of the FAA to ensure a testing process that accurately evaluates applicants' understanding of relevant material," Blair said. "We'll continue to offer our assistance and resources to support this effort, and disseminate information about future changes to the flight-instructor community."

FMI: www.faa.gov, www.nafinet.org