Addressing a Critique of the PayScale Gender Wage Gap Study

A contributor to Skepchick.org wrote an opinion piece questioning the methodology of the recently released PayScale Gender Wage Gap report. In her blog post, titled “For-Profit Wage Comparison Website Overturns Feminist Dogma. Or Not.”, Jamie Bernstein brings up several points of contention about how we arrived at our main conclusion that the wage gap is much narrower than the oft quoted 80 cents to the dollar. We regularly receive questions about our gender wage studies as our findings do not conform to the wage gaps regularly touted. We welcome these questions because it gives us an opportunity to provide insight into the interesting trends that our data reveals.

In this blog piece, I want to address Bernstein’s concerns, as well as point out some fallacies in her conclusions.

First, for those unfamiliar with our Gender Wage Gap Report, some background. Utilizing pay data PayScale collected from several million users over the last year, our team of data scientists examined the pay differences between men and women across several laterals, such as industry, job level and job title. For this blog, we are going to focus on the per-job analysis.

We started by comparing all men and women in a given job title. However, men and women are inherently different and thus make different career choices and this strict comparison doesn’t accurately capture all of these differences.

Therefore, the next thing we do is determine the typical characteristics of the male sample and the female sample and highlight where they vary. For example, women tend to have fewer years of experience. Our compensation model understands how pay changes with accumulated experience. Therefore, using our model, we adjust the experience level of the typical woman to match that of the typical man.

We did this for all compensable factors we measure, such as degree, location, industry, management responsibilities, etc. in an effort to provide an apples-to-apples comparison. This comparison provides an answer to the question, if we compare a man and a woman doing the exact same job, in the same location and industry, with the same experience level and education, etc., what pay gap still exists?

What did we find? Well, we found the pay gap largely disappears for most jobs, but does subsist for executive-level positions. We examined hundreds of jobs, which we provided to outside media outlets, but decided to highlight only 12 on our own site as an editorial choice.

Now let’s get back to Bernstein’s blog post. She first contacted me on Twitter to ask clarifying questions about our methodology. These questions were the standard, “How do you collect your data? Who was included? What are some demographic details?”

Using the limited character space, I responded the best I could to each of her questions, even though she claims otherwise: “I wrote to the author Katie Bardaro on Twitter to see if I could get a little bit more information about the people surveyed…PayScale doesn’t seem to offer up this information and Katie didn’t reply to my follow-up questions, so we’re just going to have to guess.” We’re always happy to talk with the media, but we don’t think Twitter is the ideal format. We’d encourage Bernstein, and any other media to contact us via press@payscale.com.

Bernstein then goes on to say she read our methodology and that it said simply, “According to their Methodology, PayScale surveyed 13,500 U.S. working residents. Also ….  Nope. That’s it. That’s the methodology.”

In fact, we provided detailed definitions of the measures included in the data package, as well as the sample size utilized for optional questions we added into our regular, ongoing compensation survey.

I should also mention that contrary to Bernstein’s belief (“Most of the links tend to go to pages asking me to give them money to tell me what my salary is supposed to be.”), the PayScale salary survey is completely and utterly free for our users. We crowd source data around salaries. Thus in our barter system, you provide us with information about your job, employer and background and we provide you with a free, detailed salary report that helps you to understand how you compare to others like you. In the spirit of full disclosure, we do sell compensation software to employers to help them better understand how to pay their employees using aggregated data collected via our salary survey.   

Let’s get back to Bernstein’s understanding of our methodology. The sample size of 13,500 is for a subset of questions we added to our regular salary survey temporarily, not for the entire study. Those questions were:

• Have you ever negotiated a job offer to get a higher salary or better benefits?
• Have you ever asked for a raise or promotion?
• What’s the gender of your current boss?
• Have you ever had a female boss?

I will admit that the methodology is a little vague when it comes to the sample utilized for the wage gap, so let me clear that up right now. We utilized data collected from the PayScale salary survey over the last year for the Gender Wage Gap study – this is several million data points—a sample large enough to take a deep dive into specific positions.

However, since Bernstein assumes the sample size of 13,500 is for the full report, she goes on to provide an example breakdown of how this cuts by job and as you would expect, it results in relatively small per-job samples. And thus comes her main point of contention – sample size. One of the best ways to eradicate sample size issues is with both more data and a model that can fill in the holes. Luckily, at PayScale, we had both. Not only was the analysis based on a larger sample, but we utilized our compensation algorithm to help adjust differing characteristics between the male and female samples.

Bernstein doesn’t seem to disagree with our conclusion, just about how we got there and what our motivations may be. She ends her piece by saying, “I can only conclude from all this that PayScale’s study… is actually a concealed advertisement for compensation software put together by an extremely dodgy looking company  in order to trick terrible ’news‘ outlets into writing about it.”

We’ll admit that we are a for-profit company and we generate the majority of our revenue by selling compensation software to employers, but the salary reports we provide to individuals are free. And, as a data and software company, we thrive on collecting, cleaning, analyzing and reporting on data. The data we collect reveals some pretty interesting trends, and we love to share what we have learned about compensation with a broader audience.

We know the conclusion of the Gender Wage Gap study is controversial. It goes against what all of us have heard about the difference in wages between men and women for a long time. But, we believe, and were able to prove, that previous findings about the gender wage gap didn’t dig deeply enough to tell the whole story. There are certainly still some real issues at hand when it comes to comparing men and women in the working world. Why do women pursue different careers than men? How can we encourage women to enter high-paying, in-demand STEM fields and how can we encourage employers to support women in these roles? This should be the focus of discussion and legislation. As we and many other researchers (including Bernstein) have found, the issue isn’t a wage gap, but a jobs gap.

(Photo credit: Bill & Vicky Tracey/Flickr)