Field team debrief with pollsters Joshan, Key, Flo and Zex.
Recently, Whitehead Communications shared the results of the Kampala Holiday Survey conducted in January and February of 2018 to gather market research data on how Kampala’s residents spent their December 2017 holidays. We released our results publicly because we identified a great demand for more local data, but the results were not perfect. This survey was a test case meant for our team to learn from, and we sure did learn a lot. We are sharing some of those lessons learned with you here in the hopes that more Ugandans can use this information to improve local data collection, analysis and use.
Our total number of survey respondents was 491, which according to a commonly used sample size algorithm can account for an estimated Kampala and Wakiso resident population of about 4 million, with a confidence level of 95% and a margin of error of +/- 5%. I am not a statistician, but the stories our field team brought back were full of suspicion, lies and pretence, with some people sharing too much information and good humour also. Our conclusion in debrief was that we got a lot of real answers, but on the other hand, it is not easy to go out into Kampala and get honesty from people.
Lesson # 1: Data collection methods matter
Whitehead Comm Polls gathered responses in two ways: online and on the ground, in person. We did not use telephone interviews. It became clear early on that online responses alone would not give us a true picture of Kampala residents, because the people whom we reached out to through Whatsapp, Twitter and Facebook tended to be the young elite, reporting to earn a higher income and more male than female. It was necessary to move around Kampala to different areas in order to find a more representative sample of Ugandans. This was more costly, as pollsters had to be paid for their time, surveys had to be printed, and people needed transport and airtime to operate, but it paid off when we were able to survey more Ugandans beyond the minority that is digitally literate.
Another challenging but important element of our methodology was the standardization of how we communicated our questions. We sat together in a multi-lingual group talking through the translations from English to Luganda and debating which words better represented the meaning of what we wanted to say. We also learned quite a bit about how to sequence and structure the questions in order to avoid redundancy and take people through the questionnaire in the most consistent and efficient manner possible. Indeed, we will write our questions differently next time.
Lesson # 2: Watch out for “response bias”
“Response bias (also called survey bias) is the tendency of a person to answer questions on a survey untruthfully or misleadingly. For example, they may feel pressure to give answers that are socially acceptable.” According to Statistics How To.
Over half of our surveys were conducted by the Whitehead Comm field team in person, and this experience suggested that respondents might not be entirely truthful when answering questions. We can’t be sure, but all of our pollsters had a gut feeling that respondents were not being entirely honest at least once. I suspect that pressure to meet social expectations gave some of our respondents the incentive to lie in order to save face.
Also, many respondents did not understand that their responses would be anonymous. They were not familiar with the concept of aggregated results, and believed that their information might be used for something that could blow back on them personally. Over time, we found more clear and effective ways to explain anonymity, like by saying: “we don't take your name, and once we mix yours with the other papers, nobody will know which is yours.”
Two particularly sensitive questions that may have been affected by response bias were asking whether people were sexually active and how much money they made. In person, Whitehead Comm pollsters found that after ticking “married,” people tended to be confident in saying they had sex, but after they had ticked “single,” they were more hesitant to admit to any sexual activity, probably because this highly religious society might mark a sexually active unmarried person as immoral.
Lesson # 3: Measuring income
In order to create demographic profiles of our respondents, we asked people what was their estimated income per month. It turns out that this is a flawed question, particularly in a society where a lot of work is informal – ex. many people are brokering deals, hoping for commissions if it works out, or working jobs that pay irregularly – and few people can honestly expect a standard salary to arrive each month.
This question also aroused a lot of suspicion among respondents. Some people were very open about how much money they made, but others asked if we were government spies sent to tax them! Response bias came in once again, as those who were suspicious of taxes may have low-balled their income reporting, while others made our pollsters wonder if they were just showing off. For example, one of our teammates interviewed someone they described as “a slay queen” with a “show off attitude” who said she made over five million shillings per month, and when he asked her if/how she travelled over the holidays, she said “by plane, OBVIOUSLY,” which made him wonder to whom in Uganda it would be obvious that anyone would fly abroad (according to our results, only 6.3% of respondents did).
Just after finishing the Kampala Holiday Survey and debriefing with our team, I bumped into the Deputy Governor of the Bank of Uganda, Louis Austin Kasekende, and he provided some insightful advice on how to measure incomes in Uganda. “Don’t ask people how much they make,” he said. “Ask people how much they spend.” Indeed, the money has to come from somewhere – whether it is debt or work or handouts - but expenses may be a more real and truthful number than what people say when you ask them how much they make. Whitehead Comm Polls will definitely keep Mr. Kasekende’s advice in mind next time we try to find out how much money people have.
Pollsters Joshan and Key
Lesson # 4: Prepare pollsters to have a hard time
We sent five pollsters all over Kampala to survey people, and they did not have an easy time. “I expected people to at least be nice and give me a few minutes,” said one of our team members, but she quickly learned what it is like to be viewed as an annoyance and accept a lot of rejection between collecting responses. Several people asked for money to respond to the survey, which we did not give.
One unexpected challenge was that, despite printing surveys in English and Luganda to enable people to fill them out privately, very many Ugandans preferred to be read the questions and answer them in public. It was unclear whether the respondents were illiterate or just preferred the service of being spoken to rather than writing their own responses, but either way this was not an ideal method of collecting answers, since the lack of privacy may have increased the risk of response bias. Our field team tried to direct respondents towards filling out the surveys privately, but believed that many felt a sense of entitlement to have the survey read and filled out for them.
Our female pollsters faced further challenges in their work as they were repeatedly sexually harassed by male respondents. All three of our female pollsters reported having men turn the questions back on them, especially the ones about relationship status and sexual activity, which made them uncomfortable. When we discussed how to handle these situations, female pollsters said that they didn't want to lose out on the respondent filling the entire survey, so they felt that they had to be sweet and giggle and allow men to flirt with them throughout the process in order to get their job done. This was disheartening, and just one example of the vulnerability of women and corruption of relationships in Ugandan society. We decided that pollsters would work in teams and operate only in public places in order to ensure their safety, and pollsters were advised that if they ever felt unsafe or uncomfortable, it would be better to lose the respondent than continue.
Lesson # 5: Kampalans are suspicious people
The most common challenge reported by Whitehead Comm pollsters was that they were suspected of being spies for the Government. People did not understand market research and openly voiced their suspicions, with one respondent saying "there is something that Government is planning here." In our debrief, one pollster explained a personal experience when there was an incident in a marketplace and merchants, boda drivers, etc. suddenly began whipping out pistols in response, which made that pollster believe like many respondents did that spies and undercover security operatives were everywhere, so it was only prudent to be paranoid in Kampala. Several respondents questioned why we wanted to know how much money they made, how much they spent on gifts, or what method of transport they used for travel. The survey was intentionally not political, but still politics came into it as respondents asked if we were gathering information to create new tax policy, or if we wanted to know their political opinions (which we did not ask for).
We managed this challenge by clearly representing the Whitehead Communications brand and website on survey materials, with pollsters wearing Whitehead Comm t-shirts, introducing themselves as interns and directing people towards our website to increase a sense of clarity and transparency about who we were and what we were doing.
On the other hand, despite some people being too suspicious to answer our questions, others were very free with their personal information and would begin talking all about their holiday, providing so much detail, until the pollster would have to stop them and ask that they respond to specific questions. Once pen touched paper, many of those who previously offered too much information then began to hesitate with their answers.
In conclusion, we learned a lot from conducting this survey, so in that respect the activity was a success. This article contains only a few highlights, but we hope that by sharing our experience here, other people in Uganda can get a head start in contributing to local data collection.
Is there anything we missed that you would like to share? We are so open to learning, so please share your thoughts in the comments below!
