Find Out Now Panel Methodology

Panel Methodology

Find Out Now gathers responses to market research questions from the daily visitors on the Pick My Postcode website. Pick My Postcode is a free daily “lottery” with a number of draws offering cash prizes to winning postcodes registered by the members. The questions fielded through Find Out Now appear on the website’s Survey Draw, where the members are asked to answer them before the winning postcode is revealed. They are not required to answer the question(s), nor are they compensated in any way, and the winning postcode is revealed should they choose to not answer by pressing “No thanks, not today”.

Each day ~100,000 members answer questions in this way, and most of them return every day. This makes it possible for Find Out Now to predict the number of question completions from a target audience, and manage demographic quotas on the fly. These demographic variables are asked of the members in their early visits to the Survey Draw, so that profiling can be leveraged in this way.

The number of questions presented to a member on any day is usually limited to seven or fewer, but we can make exceptions if more need to be answered quickly. Otherwise, we can split longer surveys into sets of seven or fewer questions and run them over multiple days, targeting the same respondents.

The survey questions are backfilled by a large bank of profiling questions, so that if there are no commissioned surveys available, the member has at least one question to answer each day. This profiling can be used for targeting specific audiences, and in conjunction with live screening questions, so there is no limit to the targeting capabilities.

Find Out Now is able to deliver nationally representative results in near real time by using two proprietary algorithms called Optimal Sequential Inclusion (OSI), which chooses whether to ask someone a question, and Optimal Random Elimination (ORE) which removes (at random) over-quota demographics in the final sample set. These processes are explained in more detail below.

Nationally representative results

National representation describes a sample that is representative of a population in terms of its respondents, with respect to a number of demographic variables (typically age, gender, region, and social grade).

Find Out Now surveys nationally representatively using its proprietary algorithms to ensure the aggregate demographic profile of the respondents in the survey sample is within ±1% of demographic quotas in accordance with the census data released by the Office for National Statistics.

Optimal sequential inclusion (OSI)

OSI is Find Out Now's proprietary algorithm that works in real time with Find Out Now's survey sampling data stream to ensure the collected sample adheres to pre-defined demographic quotas.

This algorithm makes real time sampling decisions based on demographic quotas. It does this by calculating the aggregated demographic deviation and only survey respondents who on average will improve the demographic balance of the sample. In effect, this algorithm can be viewed as a manager who works actively to make the sample as balanced as possible.

As the resultant sample is already very close to the pre-defined, the use of post-stratification methods is minimised. For example, terminal weights produced by raking are close to one and sample variance is not unnecessarily inflated. In addition, any over-sampling for the survey is minimised.

Optimal random elimination (ORE)

Optimal random elimination is a post-stratification method that selects a subsample which adheres to pre-defined demographic quotas from the raw sample, while maintaining the highest possible sample size.

The algorithm works iteratively. At each iteration, an imbalance score is calculated for each respondent in the sample based on the respondent's demographic profile. A random respondent with the highest imbalance score is then removed from the sample. This is repeated until all demographic quotas are deemed acceptably close to target.

Despite achieving the same purpose as traditional raking, ORE does not produce a sample with decimal numbers as weights.

Frequently Asked Questions

How are survey responses collected?

FON surveys rely on PMP members to answer questions as they visit the site. PMP members are incentivised to visit the site daily to earn bonuses and claim any giveaway winnings. They do this by participating with site activities and one of these activities is answering survey questions if they so choose. PMP therefore collects responses passively and does not actively invite respondents. The collection process runs continuously as a data stream and FON can collect up to 100,000 responses a day. Thanks to the large quantity of streaming responses that originate from different parts of the UK and various demographic backgrounds, the responses collected are a sufficiently random sample.

We also take this opportunity to ask PMP members demographic questions, such as gender, age and political preferences. Over time, each PMP member is deeply profiled and we have detailed knowledge on information that we think are going to be useful in the future for targeting surveys towards specific audiences. This demographic information is also updated on a regular basis to reflect any changes.

What is PMP and where does it fit in the picture?

PMP, short for Pick My Postcode, is the UK's biggest free daily giveaway site. It is a free to enter daily postcode draw platform available to all UK citizens. There are five daily Pick My Postcode lottery draws: the main draw, the video draw, survey draw, stackpot and bonus draw. A new winning postcode for each draw is selected every day and therefore PMP members are incentivised to visit daily.

How does Find Out Now survey specific target audiences?

Surveys that are designed to be answered by a specific target audience do not deviate from FON's passive approach. Instead, we make sure these targeted surveys are only shown to eligible respondents, who are either already profiled by another question in our database or by a new profiling question which will be shown to all PMP users.

What is “Prefer not to say” and why do we keep it in the survey results?

“Prefer not to say” is an answer option that's included in every survey question asked on Find Out Now by default. This answer option allows our panel members who are either unwilling or unable to answer. This also helps us capture the degree of non-response bias for the survey question.

Keeping “Prefer not to say” responses in the survey results reveals the survey result uncertainty caused by non-response bias. High proportion of “Prefer not to say” responses may indicate that the survey question is poorly designed or that it is asking for sensitive information (e.g. income or medical history). Poorly designed survey questions usually can be improved to minimise non-response bias.

What is non-response bias?

Non-responses bias refers to the systematic difference in survey responses between responders and non-responders. This happens when a group of respondents are more inclined to provide survey responses than another group of respondents who have a different view about the subject matter being surveyed.

Why are some of the survey responses percentages in my survey report displayed in blue?

We display survey responses percentages in blue for demographic groups if they are determined to be statistically significantly different from the aggregate total percentage.

Highlighting “Prefer not to say” percentages in blue is particularly helpful for identifying demographic groups that are more or less inclined to respond to the survey question than average.

What are statistically significant results?

In statistics, statistical significance is the probability of obtaining results as extreme as the ones observed, assuming that the null hypothesis is true. The null hypothesis is a statement of no effect or no difference. A statistically significant result is one that is unlikely to have occurred by chance alone.

The level of statistical significance is usually expressed as a p-value. A p-value of 0.05 is often used as the threshold for statistical significance, which means that there is a 5% chance or less of obtaining results as extreme as the ones observed if the null hypothesis is true.

We use a binomial t-test with a confidence interval of at least 95% to calculate the statistical significance of the data presented in our results tables.

Why are there letters next beside some results on my reports?

This is intra-demographic statistical flagging. Letters indicate a statistically significant difference from others within the same demographic group.

Each segment within a demographic group is allocated a letter from A-Z, segments that pertain any statistical significance from another are flagged with their corresponding letter.

Is Find Out Now's methodology classed as probability sampling or non-probability sampling?

Owing to the track record of good prediction accuracy, probability sampling based surveying has been crowned king in the past half century. However, its rising cost and concerns about coverage and non-responses biases means that non-probability sampling has garnered interest in recent years as a cost-effective alternative. FON does not strictly belong in either of these sampling paradigms, and surveys conducted by FON are just as robust, if not more so, as other offerings on the market.

Probability sampling refers to the sampling technique in which the researcher selects samples from the population randomly using methods based on probability theory, whereas non-probability sampling refers to the sampling technique in which the researcher does not sample randomly, but selects samples based on the judgement and domain knowledge. Non-probability sampling is often criticised for being unrepresentative and suffers from a myriad of data biases, though academic research has shown that non-probability sampling is a powerful technique when used correctly [1].

FON's methodology has characteristics from both paradigms. FON relies on members of PMP to visit daily and answer survey questions. This is akin to opt-in survey panels in which respondents are self-selected. Despite this similarity, PMP members cannot be classified as professional survey respondents, a term that is strongly associated with opt-in panels. PMP members are usually limited to only answering 1-7 questions each day. In addition, since everyone above the age of 18 in the UK can participate in PMP, the result is a data stream of sufficiently random samples that originates from different parts of the UK and from different demographic backgrounds, which rivals probability samples in terms of being representative.

[1] Zhang, C., Antoun, C., Yan, H.Y. and Conrad, F.G., 2020. Professional Respondents in Opt-in Online Panels: What Do We Really Know?. Social Science Computer Review, 38(6), pp.703-719.

What if my question isn't covered here?

If you have any questions that aren't covered here, please email our customer service team at: ask@findoutnow.co.uk