How Large a Salary Survey Sample is Enough? (II)

November 3, 2006

In a previous post, I claimed that as few as 5 employee profiles are enough to report accurately on the typical pay for a job. How can that be?

In this post, we’ll look at how statistics work, and why a small, targeted, data set is often preferable to a much larger, but poorly characterized one. You don’t even need fancy math to calculate this.

If you are curious what kind of sample we have for your job, try the PayScale salary calculator.

Washington State Employment

For example, in Washington State we have hundreds of Warehouse Laborer profiles. The typical pay range is $11.61 to $15.21, and PayScale reports a typical hourly wage of $13.13. If we had used only a sub-sample of 5 profiles (as opposed to hundreds of profiles), statistical fluctuations would have lead to us report a typical hourly wage between $12.00 and $14.50, or about +/-10% of the true (population) value, 99% of the time.

Do You Know What You're Worth?

How bad a mistake is an answer of, e.g., $14.50, the upper limit of what might be reported in this small sample of 5? First of all, $14.50 is well within the typical range of wages (25th to 75th percentile given above) for Warehouse Laborers. It is one of the typical wages that most Warehouse Laborers in Washington State are paid.

Is the difference between $14.50 (small sample pay) and $13.13 (true population pay) significant? Absolutely. Is it a mistake to pay Warehouse Laborers an hourly wage of $14.50? No. For example, $14.50 is the typical pay for Warehouse Laborers with 5 to 9 years of experience working in Seattle.

Hourly Wage in the city of Seattle

The reason the state-wide pay has such a broad typical range (25th to 75th percentiles), is because we did not included some of the most important factors: the level experience of the Warehouse Laborer, and the location of the job. To determine average pay, getting these factors right is critical.

Let’s consider now only experienced Warehouse Laborers in the city of Seattle. The range of true (population) wages is much narrower than for the state-wide data, because we have included all the important factors that determine pay. Even a small sample of 5 profile would give an hourly wage between $13.75 and $15.25, or about +/-5% around the true value of $14.50, 99% of the time.

The problems I am ignoring for this post – like survey bias, worker effectiveness, business needs, etc. – are all likely to cause larger variations in pay than this 5% statistical uncertainty in this "average" pay. While we generally prefer more than 5 profiles in search results at PayScale, it is not because of the statistical fluctuations in such a small sample.

Finding Average Pay without Fancy Math

This brings us to the crux of the PayScale system. PayScale produces a better answer for what a job pays by matching to the complete description of each job. Even if we have fewer profiles in a particular match, the answer will be much more accurate than one based on thousands of employees scattered across the nation with different skills and experience.

We have thousands of Warehouse Laborer profiles across the US, and we do use them all to narrow in on the specifics of a particular job. However, rather than try to extrapolate from a broad range using fancy math to get the pay for a specific job, we prefer to search for the sub-set of data where all the important factors match the specific job requirements.

Typical Pay, Hundreds of Profiles

Of course, we don’t use only 5 profiles in the vast majority of reports. Typically we have dozens or hundreds of profiles that match, even after specifying all the important criteria. Since they are well-matched, we use them all, even though the reduction in statistical uncertainty for the "typical" pay is not significant, a few more data points are useful for reporting the typical range of pay.

I must admit, I am also a little amazed that as few as 5 well-matched profiles can still give useful information about what typical pay is, but the mathematics of statistics does not lie.

Looks like I will need a future post to discuss how much data we need to answer "typical pay range" questions. In the meantime, figure out whether you are being paid the typical amount by completing our salary survey.

Cheers,

Dr. Al Lee