SAT Sampling Bias: Why a Huge Sample Can Still Be Bad

A large sample is not automatically a good sample.

If the sampling method systematically overrepresents some kinds of people and misses others, increasing the sample size may make the estimate more precise about the wrong group.

The key distinction is:

• sample size affects random sampling variability and margin of error;
• sampling method affects whether the sample is representative of the population.

A random sample gives each population member an appropriate chance to be selected and supports generalizing sample results to the population the sample was drawn from.

A biased sampling method can fail even with thousands of responses.

Why sampling bias is directly relevant to the SAT

College Board's Problem-Solving and Data Analysis domain includes inference from sample statistics and evaluating statistical claims. Its current specifications state that students should understand which population results from a random sample can be extended to.

So SAT sampling questions can ask:

• whether a sample is representative;
• which population a result can be generalized to;
• whether a survey has selection bias;
• whether a larger sample fixes a design problem;
• how sample size affects margin of error;
• whether the sample source matches the target population.

For the broader inference topic, see SAT Margin of Error and Sampling.

Population vs sample

Population

The full group you want to learn about.

Example:

all students at a high school.

Sample

The smaller group actually measured.

Example:

120 students selected from the school.

The goal is often to use the sample statistic to estimate a population parameter.

That inference is only credible if the sample selection supports it.

What makes a random sample useful?

A random sample uses a chance-based process designed to avoid systematic preference for particular members of the population.

Example:

A school has a complete list of 2,000 students. A computer randomly selects 200 distinct students.

That sampling process is much more defensible for estimating school-wide opinions than asking the first 200 students entering the gym.

Random sampling does not guarantee the sample exactly matches the population. Random samples still vary.

But the variation is due to chance rather than a built-in selection mechanism.

Worked example 1: convenience sample

A principal wants to estimate how satisfied all students are with cafeteria food.

She surveys 150 students eating in the cafeteria during lunch.

What is the main problem?

Students who choose to eat in the cafeteria may differ systematically from students who bring lunch or leave campus.

The sample may overrepresent students willing to use the cafeteria.

The sample may be biased and not representative of all students.

The issue is not that 150 is "too small" by itself.

The issue is who was eligible to be sampled.

Convenience sampling

A convenience sample chooses people because they are easy to reach.

Examples:

• surveying people in one hallway;
• polling only customers currently in a store;
• asking students in one class about the entire school;
• sampling people at one location and time.

Convenience is not the same as randomness.

A large convenience sample can still be systematically biased.

Worked example 2: huge but biased sample

A website asks visitors:

"Click here if you want to rate our new design."

It receives 50,000 responses.

Can those responses automatically be generalized to all users?

No.

People who choose to respond may have stronger opinions than typical users.

This is voluntary-response bias.

$$\boxed{\text{Large }n\text{ does not remove self-selection bias.}}$$

Voluntary-response bias

In voluntary response, participants choose whether to enter the sample.

People with strong positive or negative opinions may be more likely to respond.

Examples:

• online polls anyone can click;
• call-in surveys;
• optional feedback forms;
• social-media polls.

The sample can be large while still unrepresentative.

Undercoverage

Undercoverage occurs when part of the target population has little or no chance to enter the sample.

Example:

A city wants to estimate residents' internet access and conducts a survey only through an online portal.

People without reliable internet are exactly the group most likely to be excluded.

The method can understate the lack of access.

Worked example 3: undercoverage

A university wants to estimate how often students use the campus library.

It samples only students enrolled in humanities courses.

Why might this be biased?

Students in other majors may have different library-use patterns and are underrepresented or excluded.

The appropriate population is:

all university students,

but the sample frame covers only a subset.

Nonresponse bias

Even if people are randomly selected, not everyone responds.

If nonresponders differ systematically from responders on the topic of interest, the final responding sample can be biased.

Example:

1,000 randomly selected households receive a long financial survey. Only 200 respond.

If respondents differ systematically in income, financial literacy, or willingness to disclose financial information, the response set may not represent the original random sample well.

The SAT may not always name "nonresponse bias," but it can test the underlying reasoning.

Wording bias is different from sampling bias

Suppose the sample itself is random, but the question asks:

"Do you support the wasteful new school policy?"

That wording may influence responses.

This is a measurement/question-wording problem, not a sampling-selection problem.

Keep separate:

• who was sampled;
• how the question was asked.

Both can damage a study, but in different ways.

Bigger sample vs better sample

Suppose Survey A randomly samples 400 students from the full school roster.

Survey B asks 4,000 followers of the school's athletics social-media account.

Which is more defensible for estimating opinions of all students?

Survey A.

Why?

Survey B may heavily overrepresent students interested in athletics or social media.

A sample of 4,000 can be worse than a random sample of 400 if the selection mechanism is biased.

Sample size and margin of error

For a properly designed random sample, larger sample size generally reduces sampling variability and leads to a smaller margin of error.

College Board explicitly expects students to understand that relationship.

But:

larger sample size does not automatically fix sampling bias

This distinction is crucial.

See SAT Margin of Error and Sampling.

Worked example 4: what a bigger random sample changes

Survey A:

• random sample;
• $n=100$.

Survey B:

• random sample from the same population;
• $n=1,000$.

All else equal, Survey B generally has a smaller margin of error.

But neither random sample is guaranteed to match the population perfectly.

The larger sample reduces chance variation, not every possible error source.

Generalization: where can the result apply?

Suppose a random sample of students is drawn from one high school.

The result can reasonably be generalized to:

students at that high school

assuming the sampling method is appropriate.

It cannot automatically be generalized to:

• all students in the city;
• all teenagers in the state;
• all U.S. students.

The sampling population sets the boundary of the inference.

Worked example 5: population boundary

Researchers randomly sample 500 registered voters in County A.

They find 62% support Proposal X.

Which population is directly supported?

Best answer:

registered voters in County A.

Not all adults in the state.

Not all residents of the country.

The sample frame determines the target population.

Census vs sample

A census attempts to measure every member of the population.

If complete and accurate, it avoids sampling variability because no sample is being used.

But censuses can still suffer from:

• nonresponse;
• measurement error;
• missing people;
• inaccurate reporting.

"Not a sample" does not mean "immune to all bias."

Stratified sampling intuition

Sometimes a population contains important subgroups, and a random process samples within those groups.

Example:

A school is:

• 40% Grade 9;
• 30% Grade 10;
• 20% Grade 11;
• 10% Grade 12.

A study may sample students from each grade in a deliberate way to ensure representation.

The SAT is more likely to test whether the design reasonably represents the target population than require formal sampling-method vocabulary beyond random sampling.

Sampling with replacement vs without replacement

For most SAT inference questions, the important issue is whether the selection method is random and representative, not the fine distinction between replacement rules.

Do not overcomplicate a sampling-bias question unless the replacement detail changes the probabilities.

Worked example 6: survey at one time of day

A transit agency surveys riders on buses from 10 a.m. to noon to estimate the opinions of all daily riders.

Potential problem:

Morning commuters, evening commuters, students, and late-shift workers may be underrepresented.

The sampling window creates undercoverage by time.

Increasing the number surveyed from 500 to 5,000 during the same time window does not fix that structural gap.

Worked example 7: random sample from the wrong population

A company wants to estimate satisfaction among all customers.

It randomly samples 1,000 members of its premium loyalty program.

The sample is random within premium members, but the target population is all customers.

So the results can support inference about:

premium-program members

more directly than all customers.

"Random" is only useful relative to the population from which the random sample was drawn.

Random sampling vs random assignment

These are different ideas.

Random sampling

Used to select participants from a population.

Supports generalization to that population.

Random assignment

Used after participants are in a study to assign treatments.

Supports causal comparison by balancing confounding variables across treatment groups.

Do not substitute one for the other.

See Random Sample vs Random Assignment.

Sampling bias vs confounding

Sampling bias asks:

Is the sample representative of the target population?

Confounding asks:

In a comparison of groups, could some other variable explain the observed difference?

A study can have:

• a representative sample but poor causal design;
• a strong randomized experiment using a nonrepresentative volunteer sample;
• both;
• neither.

The claims you can make depend on both selection and assignment.

Worked example 8: identify what is actually supported

Researchers randomly sample 600 adults from City Z and record exercise habits and blood pressure.

They find more exercise is associated with lower blood pressure.

What can they say?

Because the sample is random from City Z, the observed association may be generalizable to adults in City Z.

But because exercise was observed, not randomly assigned, the study does not by itself establish that exercise caused the difference.

Sampling and causation are separate questions.

Common trap 1: large sample = unbiased

False.

Large samples reduce random variability but can preserve or amplify systematic selection bias.

Common trap 2: random assignment = representative sample

False.

Random assignment concerns treatments within the study; it does not determine how representative the participants are.

Common trap 3: random sample = causal proof

False.

Random sampling supports generalization, not causation.

Common trap 4: sample from one subgroup generalizes to the whole population

Only if the target population is actually that subgroup or the sampling design justifies broader representation.

Common trap 5: "random" appears somewhere, so everything is valid

Ask:

• What was randomized?
• Selection?
• Treatment assignment?
• Something else?

Common trap 6: increasing n fixes undercoverage

It does not if the same population segment is still excluded.

Common trap 7: confuse sampling bias with question wording

Selection and measurement are different error sources.

A claim-strength routine

When the SAT describes a sample, ask:

1. What is the target population?
2. From what population was the sample actually drawn?
3. Was selection random?
4. Could important groups be undercovered?
5. Was participation voluntary?
6. Is sample size the issue, or is sampling method the issue?
7. What population can the result reasonably generalize to?

SAT-style practice

Question 1

A university surveys the first 500 students entering the recreation center about exercise habits.

Best concern?

Convenience sample may overrepresent students who use the recreation center.

Question 2

An online poll gets 100,000 voluntary responses.

Does the large response count ensure representativeness?

No

Self-selection can still bias the sample.

Question 3

A random sample of 400 students is drawn from School A.

Which population is the natural target for generalization?

Students at School A

Question 4

Two random samples from the same population have sizes 200 and 2,000.

Which generally has smaller margin of error?

$$\boxed{n=2000}$$

assuming similar design.

Question 5

A random sample is selected, but only 20% respond. Why might this matter?

Nonresponse can bias the responding sample if responders differ from nonresponders.

FAQ: SAT sampling bias

Does a larger sample always mean a better study?

No. Larger random samples generally reduce margin of error, but a large biased sample can still misrepresent the population.

What is a random sample?

A chance-based selection from a population designed to avoid systematic selection preference.

What is convenience sampling?

Selecting people because they are easy to reach rather than through a representative random process.

What is voluntary-response bias?

Bias that can arise when people choose themselves into the sample, often because they have stronger opinions.

What is undercoverage?

When some parts of the target population have too little or no chance to be sampled.

What does random sampling let me do?

It supports generalizing results from the sample to the population from which the sample was randomly drawn.

Does random sampling prove causation?

No. Causation is tied to experimental design and random assignment, not merely random sampling.

How does sample size affect margin of error?

For comparable random samples, larger sample size generally produces a smaller margin of error.

How to practice this on MastaPrep

MastaPrep starts with a short placement check, then builds a study plan around the SAT Math skills where you are losing the most points. If this one keeps costing you questions, it shows up in your plan rather than getting lost in a random practice set. Every question has a worked explanation, and the ones you miss come back for review a few days later, which is when you find out whether the fix has actually stuck.

See how MastaPrep works or try free SAT practice questions. The free plan needs no credit card.

Related MastaPrep guides

• SAT Margin of Error and Sampling
• SAT Probability and Statistics
• Random Sample vs Random Assignment
• Observational Study vs Experiment
• SAT Two-Way Tables
• SAT Percentages

Final takeaway

A good sample is not defined only by how many observations it contains.

Ask:

Who had a chance to be selected?

Random sampling supports population inference.

Larger random samples usually improve precision.

But a biased sampling frame, convenience sample, or voluntary-response process can remain biased no matter how many responses it collects.

---

Official SAT references:
https://satsuite.collegeboard.org/sat/whats-on-the-test/math/types/problem-solving
https://satsuite.collegeboard.org/practice/student-question-bank/math

Trademark note: SAT® is a registered trademark of College Board. MastaPrep is independent and is not affiliated with or endorsed by College Board.