The science of rating a Philippine public official

enablePagination: false
maxItemsPerPage: 10
totalITemsFound:
maxPaginationLinks: 10
maxPossiblePages:
startIndex:
endIndex:

The purpose of public surveys is to record firsthand public opinion on contemporary and pressing issues. How do Philippine survey firms do it? Illustration by JL JAVIER

Manila (CNN Philippines Life) — Last month, President Rodrigo Duterte was reported to be the “most decisive” public official in government. The headlines reflected the bi-monthly Pahayag national survey conducted by Publicus Asia, Inc. in which respondents were asked to rate the current holders of the five highest executive positions in the government according to the firm’s “Love, Care and Solidarity Index,” as well as its “Decisiveness Index.”

According to the survey, which had a ± 2.58-percent margin of error, President Rodrigo Duterte topped both categories, earning a 90 percent rating for decisiveness, 82 percent for love, and 79 percent for care and concern. Vice President Leni Robredo, meanwhile, earned a five percent rating for decisiveness, 13 percent for love, 17 percent for concern, and 16 percent for care. The rest of the personnel — Chief Justice Maria Lourdes Sereno, Senate President Aquilino Pimentel, and Speaker Pantaleon Alvarez — did not fare much better.

The survey was met with criticism following sensationalized reporting and allegations of questionable methods, such as the use of metrics that are more difficult to measure empirically than standard approval ratings.

“If you have the freedom of speech, there’s also the freedom to listen, and the freedom to inform the public what other people are thinking.” — Leo Laroza

The purpose of public surveys is to record firsthand public opinion on contemporary and pressing issues, says Leo Laroza, communications and IT director at Social Weather Stations (SWS). “Our main focus is on subjective quality of life,” he adds.

For example, in SWS, governance, democracy, poverty, crime, human rights, and inflation are some of the topics that the institution covers when it conducts surveys. Recently, they have also been asking respondents about their thoughts on the possible nationwide implementation of martial law and extrajudicial killings.

In an interview with CNN Philippines Life, Laroza detailed the work that goes into the surveys conducted by SWS, including the tools and methods they use, as well as the measures they take to ensure fairness and accuracy.*

Formulating the questionnaires

Every quarter of the year, SWS selects 10 specific topics through which respondents can evaluate the current administration. “Every new question that gets formulated has to go through a series of pre-testing,” says Laroza. Test respondents representing different economic classes and educational attainments are gathered to ensure that a question will be understood by every person, regardless of background.

Measures are also taken to avoid biases or different interpretations, such as thorough quality control and editing. “We cannot allow leading questions, biased questions, or overly sensitive questions,” Laroza says. “We look at sequence bias. We cannot, for example, start talking about sensitive things and then ask a series of questions about, ‘How satisfied or dissatisfied are you with the government?’”

Determining the sample size

A sample size of 1,200 respondents is allotted for every survey, as determined by funding and statistical data. Field staff are dispersed across four major areas: Metro Manila, Luzon, Visayas, and Mindanao. All municipalities of Metro Manila are typically included; for the rest of the areas, locations are selected based on the adult population. For each major study area, 300 responses must be collected over the course of three days.

“The function of [the] sample size depends on [how large] an error margin you are willing to work around [with],” Laroza says. The current sample size of SWS makes for an error margin of ± 2 percent on the national level, and ± 6 percent on the major area study level.

Surveys-02.jpg SWS continues to conduct surveys, Leo Laroza of SWS says, because “we want to keep a record of Philippine history as [it] unfolds from the point of view of the people themselves.” Illustration by JL JAVIER

Pulse Asia, another survey firm, also typically makes use of a sample size of 1,200. According to their website, this sampling results in a ± 3 percent error margin on the national level and ± 6 percent on the subnational level. Publicus Asia, meanwhile, has a usual sample size of 1,500, making for a ± 2 percent margin of error on the national level and ± 3 percent for the areas of study.

Selecting respondents

Since only one respondent can be interviewed per household, a Kish grid — a method in which a pre-assigned table of numbers is used to generate a random selection — is used to determine the qualifying members. Each respondent must be at least 18 years old; when an odd-numbered questionnaire is in use, the respondent must be male, and if it’s even-numbered, the respondent must be female.

To avoid bias, the field staff conducting interviews in barangays must not be residents of the areas they visit, and these locations are kept confidential to avoid opportunism, especially during election season.

Properly orienting the respondents

When speaking to potential respondents, the field staff from SWS introduce themselves using a spiel which Laroza says is guided by a worldwide code of professional ethics from the World Association for Public Opinion Research. “It should be clear in any interview activity when you establish your interviewer-correspondent relationship that his or her answers will be confidential and that he or she will not be directly identifiable to whatever she says,” Laroza explains.

Aside from the respondents’ privacy, he adds that it’s important to reassure them that the process is completely voluntary and that there are no right or wrong answers. “If they start feeling uncomfortable about the questions, then at any point in time, they can also refuse.” Non-answers are also part of the analysis; they take note of which questions made respondents uncomfortable, for example, or which questions they didn’t know the answers to.

Conducting the interviews

SWS surveys are conducted face-to-face and still make use of pen and paper interviews. Laroza explains that they stick with this method because it’s reliable, based on trust, and still the “best method when you’re dealing with a lot of topics.” In comparison to SWS’ four-day process of data collection, Pulse Asia’s surveys take six days to complete. They also make use of face-to-face interviews.

Publicus Asia also takes three days to collect data. A major difference, however, is that instead of pen and paper, they use a mobile survey app to conduct interviews. According to their website, computer-assisted personal interviews remove the need for data encoding and can be processed more quickly. Results can also be delivered in real-time.

Analysis and interpretation

Laroza says that SWS often works with a media partner to disseminate information. In trying to interpret the given data and analysis, reporters, he says, must take note of when the survey is done, who its respondents are, how big the sample size is, and what the assigned error margin is.

Surveys, of course, are based on opinion; however, it’s important that these opinions are formed with critical thinking, a complete grasp of the situations and facts at hand, as well as ways to objectively measure certain variables.

“The practice of public opinion surveys really took off when it found its freedom to do so after the Marcos dictatorship,” he says. Public surveys may have existed at the time, but the task of informing the people was hampered, or even outright banned. Since its rise following the end of the regime, the national survey “has become an integral part of democracy.” He adds: “It’s not an answer-all to any question. It’s just a snapshot of a certain point in time.”

How do new metrics fit in?

In nationwide surveys conducted by Publicus Asia on the Psychographics of Filipino Voters in 2009 and 2015, the firm introduced variables relating to love, care, and solidarity as metrics in an attempt to, according to their website, “bring Emotional Quotient (EQ) into the way leaders are measured in polls.” The article points out that there are “no complicated mathematical computations” in these metrics, only that they can be determined based on the subject’s “self-regulation, motivation, empathy, and social skill.”

The “Decisiveness Index,” on the other hand, is an all-new metric that “measures [the] perceived decisiveness of leaders,” or their ability to make speed and sound decisions. “Awareness and approval ratings are standard survey metrics,” it says on the Publicus Asia website. “Decisiveness [wasn’t] being measured until [the] Pahayag poll did.” The firm claims that they have plans to continue using these metrics to track changes in the data as the months progress.

Laroza mentions that surveys are guided by scientific methods and empirical data. He also reiterates the importance of wording and avoiding loaded questions.

“Without mechanisms like this, how else can people in the position to make policies know how the people feel or what the people are thinking?”

Surveys, of course, are based on opinion; however, it’s important that these opinions are formed with critical thinking, a complete grasp of the situations and facts at hand, as well as ways to objectively measure certain variables.

Why keep conducting public surveys?

Nowadays, the ability to speak one’s mind is a right: freedom of speech is guaranteed by the Constitution, and the practice of public opinion essays is protected by the Supreme Court. “If you have the freedom of speech,” Laroza says, “there’s also the freedom to listen, and the freedom to inform the public what other people are thinking.”

SWS continues to conduct surveys, Laroza says, because “we want to keep a record of Philippine history as [it] unfolds from the point of view of the people themselves.” He encourages people, in turn, to answer with honesty and sincerity. “Without mechanisms like this, how else can people in the position to make policies know how the people feel or what the people are thinking?”

***

*Editor’s note: CNN Philippines Life also attempted to contact survey firms Pulse Asia and Publicus Asia for this article, but they were not able to reply within reasonable notice until publication. More information can be found in their respective websites: pulseasia.ph and publicusasia.com.

The article first indicated that SWS' process of data collection takes three days. Laroza has clarified that the process takes four days. The updated article reflects this correction.