E-commerce
October 2, 2026
Wondering how to determine the sample size for a reliable survey? The answer lies in a formula: the more representative your sample, the more solid your e-commerce decisions are. Specifically, you need to combine your population size, the desired confidence level, and the accepted margin of error.
The trap? Underestimating the actual response rate, which is often around 15% for external surveys. You then risk having to contact more visitors than you actually get in a month. A poor estimate biases your results and your strategic choices.
So how do you determine the sample size for a reliable survey? On the agenda:
What formula should you use to calculate a representative sample?
How do you integrate the response rate into your calculation?
Why do demographics distort your results?
How do you verify that your number of responses is sufficient?
What method should you adopt when the ideal sample is unattainable?
Let's get started.
Summary
What formula should be used to calculate a representative sample?
To obtain actionable results, your sample size depends on four parameters: the total population (N), the confidence level (z), the margin of error (e), and the expected proportion (p).
The classic formula is as follows:
N: the total number of people concerned, for example your unique monthly visitors or the size of your email list.
z: the confidence level, usually set at 95%, which corresponds to a value of 1.96.
e: the accepted margin of error, often 5%.
p: the expected proportion of responses; if you do not have an assumption, use 50%.
This formula may seem complex, but it is the foundation of any reliable survey. Without it, you risk making decisions based on unrepresentative data. In practice, many online tools automatically calculate this size, but understanding the parameters allows you to adjust them according to your context. For example, if your population is very homogeneous, you can reduce the required size, whereas a heterogeneous population will require a larger sample. The key is to always document your assumptions to be able to justify your choices.
Convert over 2,000 customers on average per month with Qstomy.
The world’s 1st Shopify AI dedicated to customer conversion
Empowering 200+ e-commerce merchants
What is the number of responses required for a reliable survey?
Let's take a concrete example for an e-commerce site. If you have 1,000 unique visitors per month, a 95% confidence level, and a 5% margin of error, you will need 278 responses for your survey to be statistically valid.
This figure of 278 is the minimum number of respondents. It does not take into account the response rate, i.e., the proportion of visitors who will agree to answer your questionnaire.
For an external survey, this rate is often 15%. To obtain 278 responses, you would then need to ask 1,850 people, which is more than your total monthly traffic. Two options are available to you: extend the collection over several months or reduce your confidence level. A third option is to use incentives, such as a discount or a voucher, to increase the response rate. However, these incentives can introduce selection bias, as they attract respondents motivated by the reward. It is therefore crucial to weigh the pros and cons before putting them in place.
How do you integrate the response rate into your calculation?
The response rate is the most difficult parameter to estimate. It depends on your audience, the distribution channel, and the length of the questionnaire.
If you only survey a portion of your visitors, for example 20%, your confidence level drops. Your results may then be biased by a specific group, such as unhappy customers who are more willing to respond.
To avoid this bias, incorporate a realistic response rate from the start. If you are aiming for 278 responses with a 15% rate, plan to send your survey to 1,850 people. If your audience is more engaged, the rate can reach 30%, reducing the number of solicitations needed. To improve the response rate, personalize your invitations, send reminders, and keep the questionnaire short. One study shows that surveys under 5 minutes get significantly higher response rates. Also test different channels: email, website pop-ups, or social media, to find which one works best for your audience.
Why do demographics skew your results?
A sample can be numerically sufficient but qualitatively biased. This is the case when respondents do not represent your typical clientele.
For example, if you survey your database and 80% of the responses come from women, while your goal is to understand the preferences of both men and women, your results will be skewed.
To correct this imbalance, you must send a follow-up survey only to the underrepresented group. In this example, if you received 800 responses from women and the response rate for men is 10%, you will need to reach out to an additional 6,000 men to obtain 600 male responses and balance the sample. This step may seem costly, but it is essential to ensure the validity of your conclusions. An alternative is to weight the results, but this assumes you know the exact composition of your population. In any case, document your steps to ensure the transparency of your methodology.
How to verify the representativeness of your sample?
Demographic data is essential for verifying the representativeness of your sample. Without it, you cannot know if your respondents match your target audience.
If you do not have demographic information about your population, third-party providers can provide it based on email or physical addresses. This step is often overlooked, but it prevents erroneous conclusions.
Let's take a simple case: you want to know your customers' favorite colors. If 80% of the responses come from women, your decisions on product ranges will be biased by a single segment. Rebalancing the sample is therefore essential before taking action. To collect this data, ask demographic questions at the beginning of the questionnaire, but make sure to comply with data protection regulations. You can also cross-reference your data with your CRM data to get a more complete picture. Finally, remember that demographics are only one aspect: purchasing behaviors are just as important.
How do you check that your number of responses is sufficient?
Once the responses are collected, how do you know if they are sufficient to make a decision? The method consists of simulating the impact of additional responses.
Imagine you have 1,000 responses to a question with a distribution of 30%, 45%, and 25% among three options. If you survey an additional 100 people and they all choose option A, the distribution becomes 36%, 41%, and 23%.
This variation probably does not change your decision. By assigning 100% of the responses to a hypothetical option, you test the robustness of your conclusions. If the result remains stable, your sample is sufficient. This technique, called sensitivity analysis, is simple to implement with a spreadsheet. It allows you to visualize the potential impact of extreme responses. If you find that an option could change rank, it is prudent to increase the sample size or dig into the reasons for this uncertainty. Ultimately, the goal is to make informed decisions, not to aim for statistical perfection.
What method should be adopted when the ideal sample is inaccessible?
Rather than focusing solely on sample size, ask yourself what the goal of the survey is. If the responses obtained are actionable and consistent with your business, you have probably surveyed enough people.
A qualitative survey, with open-ended questions, can reveal a major issue with just a few respondents. Sometimes, hearing from three dissatisfied customers is more useful than a thousand responses from satisfied customers.
Also, remember that third-party data can replace a survey. Behavior analytics, customer reviews, or support data can answer your questions without additional collection. For example, if you see an increase in refund requests, you don't need a survey to know something is wrong. Use these signals to guide your actions. By combining multiple data sources, you get a more complete and reliable view of the customer experience, while saving time and resources.
Which confidence level should you choose for your survey?
The confidence level is a trade-off between precision and effort. A 95% level is the standard, but it requires a larger sample size than a 90% level.
If your population is small or your response rate is low, reducing the confidence level can make the survey feasible. You then accept a slightly higher risk of error, but you obtain actionable data.
For an e-commerce, a 90% level is often acceptable for operational decisions like redesigning a product page. For major investments, aim for 95% or more. It is important to understand that the confidence level does not measure the quality of your sample, but the probability that your results reflect reality. A 95% level means that if you repeated the survey 100 times, 95 times the results would be within the margin of error. Choose your level based on the impact of the decision: the more costly the decision, the more certain you want to be.
What margin of error should you accept for your results?
The margin of error determines the accuracy of your results. A margin of 5% means that the actual result can vary by plus or minus 5 points.
A lower margin, such as 3%, increases the required sample size. Conversely, a margin of 10% reduces the number of responses but makes the results less precise.
For an exploratory survey, a 10% margin may suffice. For a strategic decision, such as launching a new product, aim for 5% or less. Adapt the margin to the stakes of the decision. For example, if you want to estimate the overall satisfaction rate, a 5% margin is reasonable. But if you have to decide on the color of a button, a 10% margin might be acceptable. Keep in mind that the margin of error applies to each question, not the entire survey. If you ask multiple questions, the overall accuracy may be lower than that of each individual question.
How to avoid selection bias in your surveys?
Selection bias is the main risk of online surveys. The people who respond are not representative of your entire audience.
Highly satisfied or highly dissatisfied customers respond more often. The others, the silent majority, are underrepresented. This bias distorts your results, even with a numerically correct sample.
To limit it, compare the characteristics of the respondents with those of your customer database. If a significant gap appears, such as an overrepresentation of an age group, launch a targeted campaign to rebalance the sample. You can also use weighting techniques to adjust the results, but this requires reliable data. Another approach is to diversify your distribution channels: do not send only by email, but also via pop-ups on your site or on social networks. Finally, be transparent about the limitations of your survey in your reports, so that readers can interpret the results with caution.
How does Qstomy replace surveys with behavioral data?
Qstomy, the Shopify AI agent, helps merchants transform customer feedback into concrete actions. By analyzing support conversations and reviews, it identifies recurring topics without requiring surveys.
The tool guides visitors toward making a purchase with personalized recommendations, reducing the need for satisfaction surveys. It also tracks packages and manages customer service requests, providing valuable data on the customer experience.
With more than 100 merchants supported, Qstomy replaces surveys with real-time behavioral data. You get actionable insights without the constraints of sample size. For example, if many customers ask questions about shipping, Qstomy detects it and you can improve your FAQ page. This proactive approach allows you to resolve issues before they become critical. Additionally, the collected data is more objective than survey responses, as it reflects actual customer behaviors.
Which points should you check before launching your survey?
Before launching your survey, check these points:
Population: Have you precisely defined your target audience?
Formula: Have you calculated the sample size using N, z, e, and p?
Response rate: Have you anticipated a realistic rate of 15% to 30%?
Representativeness: Do your respondents reflect your customer base?
Robustness: Do your conclusions hold up against hypothetical answers?
In brief
The ideal sample size depends on your population, level of confidence, and margin of error. Incorporate a realistic response rate and check demographic representativeness. If the sample is unreachable, focus on behavioral data.
FAQ
What is the minimum size for a reliable survey? For 1,000 monthly visitors, 278 responses are sufficient with a 95% confidence level and a 5% margin.
What should I do if the response rate is too low? Extend the collection period over several months or reduce the confidence level to 90%.
How do I know if my sample is biased? Compare the demographic data of the respondents with that of your customer database. A significant gap indicates a bias.
To go further: Why use social media for e-commerce customer service? - Qstomy, Pre-purchase questions in e-commerce: 30 objections to address on your site - Qstomy, AI Chatbot for size guides: reducing returns in fashion e-commerce - Qstomy, Can you use Wix for an e-commerce site? - Qstomy, How to deploy digital marketing on an e-commerce site? - Qstomy, How to create a payment gateway for an e-commerce site? - Qstomy, How to calculate the e-commerce conversion rate? - Qstomy.
Enzo
October 2, 2026
