A simple questionnaire with a big flaw. Image from TSA Pre✓
by George Taniwaki
I recently received a voice mail message from the Transportation Security Administration. A woman’s voice told me that my Known Traveler Number (KTN) would be expiring soon and that I would need to renew it if I wanted to remain in the the TSA Pre✓ program. That’s the short line through security at the airport.
I haven’t been to the airport recently (and I hope you haven’t either) so I don’t know how long the lines are right now. But joining the TSA Pre✓ program is not expensive ($85 for 5 years) and has been worth it for me. So I pointed my browser to https://universalenroll.dhs.gov/ and started the renewal process.
Near the end of the process, I landed on a very unexpected page. It was a survey form asking questions about my flying habits (see screenshot at top of post). There are many problems with this survey that market research experts will immediately catch. But check out the fourth question. “How satisfied are you with your overall airport security experience?”
Geez, I hate airport security. It is intrusive, arbitrary, and time consuming. It also subjects you to radiation and chemicals of unknown safety. I guess it would be worthwhile if it effectively stopped violence and terrorism at a reasonable cost. Unfortunately, there is no evidence of efficacy and lots of evidence that it is really expensive.
Now, how should I answer this question? There is no explanation on the page about how your response data will be used. Specifically, there is no assurance that the responses will not be associated with your personally identifiable information (PII) and only aggregated data will be provided to the TSA.
Since TSA can make your life miserable, including revoking your KTN, the safest thing to do is to tell them you love your experience with airport security. Question 4 has 10 unlabeled radio buttons with the phrases “Extremely Poor” and “Extremely Satisfied” at the ends. I decide to pick the 9th button. High but not perfect. I figured anyone picking the 10th button will also be flagged for attention as either a liar or an obsequious bootlicker.
Anyway, as a marketer you may be tempted to increase response rate to your market research survey by integrating it into a customer transaction flow. Don’t do this. Your responses will be biased.
* * * *
Update1: Revised the third paragraph to clarify that there are many other problems with this survey. Thanks to my friend and colleague Carol Borthwick for reminding me that not all readers of my blog are survey experts. Below is a list of some of the obvious errors in this survey.
- In the first question, how should one respond if you fly for both business and pleasure? And really, you fly to a destination for pleasure, you don’t fly because the experience itself is pleasurable. Almost nobody flies for pleasure, unless they are a pilot.
- In the second question, what is the TSA trying to measure? My guess is the number of times respondents are screened by TSA in a year. A round trip usually involves two waits through the TSA line. However, one should not count trips on private aircraft where you don’t go through TSA lines or flights that originate outside the U.S., even if you go through U.S. immigration at the foreign airport.
Further, if you have a connecting flight on a US domestic flight, you usually do not go through a TSA line again. If you arrive from an international flight and pass through immigration after the flight, you usually do go through TSA before boarding the next flight. - In any event, this survey was probably designed before the collapse in travel due to Covid-19. Does the TSA want to know the number of trips respondents took last year, this year (zero for me so far), or how many they would have taken if there was no pandemic. It doesn’t say.
- What’s up with those weird ranges in question 2? And which radio button should respondents select if they fly exactly 31 times a year?
- In the fourth question, notice that the wording of the two end point labels for the scale are not parallel. The low end should read “Extremely unsatisfied”. Also there are no labels for any of the intermediate points, leaving the distance between points up to the respondent’s imagination.
* * * *
Update2: Getting back to question 2, if you have a KTN, the TSA records each time you pass through security. So it should already know the actual distribution of how many times a year KTN holders pass through security. So what will it do with the survey data? Compare the response data to the actual data for accuracy? Check for lying and throw out outliers? Who knows.
Real Numeracy 
June 6, 2020 at 11:19 pm
So many consumer services are followed with a survey (which is really a census of all transactions) these days. Every doctor visit, pharmacy pickup, customer service phone call, airline reservation, airline flight, query for insurance pricing, you name it, results in an intrusive survey request. I’ve stopped responding to all of them. Most surveys seem to be poorly designed (measuring things that are unclear, or that I don’t care about), biased, or abusive. They clearly are not designed to benefit the respondent, and make legitimate consumer research work impossible.
Surveys are often used to abuse low paid customer service employees, who will be disciplined for anything less than a perfect rating. I had this explained to me twice in the same day on a trip to England last summer, both when renting a car, and later when purchasing a 10 pound SIM card for my phone. Surveys can also be abusive to customers. Whenever I need to use an auto dealership for service, which is almost always a poor experience for one reason or another, I’m hounded for a perfect score. When I refuse to answer, I receive repeated followup calls in the hope that I will relent.
I’m not sure how demeaning your front line employees and pissing of your customers is a sensible business practice for so many companies, but I chose not to participate.