Beware Flaws in Online Physician Rating Sites

This article appears in the March 2013 issue of Medicine on the Net.

Since they debuted in the mid-2000s, websites that provide user-generated ratings of physicians have become a popular destination for consumers looking for a doctor. But several recent studies suggest the websites raise more questions than answers due to a number of flaws, including questionable rating methods and small sample pools.

While researchers acknowledge that physician-rating websites serve a purpose in providing consumers with useful information, they advise would-be patients to be cautious when shopping online for a doctor, since the ratings don't necessarily reflect the clinical competency of the physicians they rate.

A grain of salt
A recent study conducted by Loyola University Medical Center and published in the Journal of Urology reviewed the ratings of 500 urologists from more than a half-dozen websites including Healthgrades.com and Vitals.com. The study found that, on average, physician scores were based on ratings from an average of just 2.4 patients, a sample size so small that it made the ratings almost irrelevant.

"Our findings suggest that consumers should take these ratings with a grain of salt," said Chandy Ellimoottil, MD, one of the authors of the study.  "These sites have the potential to help inform consumers, but the sites need more reviews to make them more reliable."

The study included a qualitative analysis of comments written about physicians on the Vitals.com website. It found that 53% of the comments were positive or very positive and that 25% were negative, with the rest of the comments rated as neutral. The comments included one observation that an elderly physician "needs to retire because he can barely walk" and another that described a recent visit as "one of the best check-ups in a long time."

The fact that many physician rating sites don't generate enough reviews is reflected in a Pew Research Center survey that found only 16% of U.S. adults look online for physician ratings and that few of those users ever post a rating on a website.

Physicians respond
Another recent study conducted by the American College of Physician Executives (ACPE) found that physicians consider the rating systems questionable at best, primarily because many sites seem to focus on experiences not related to patient outcomes and other indicators of skill and proficiency.

The ACPE study is based on 730 responses to a 21-question survey that researchers received from ACPE members in late 2012. The majority of respondents (55%) work at health systems or individual hospitals and 20% are employed by medical groups. When asked to give their opinion of online physician rating sites, only 12% of respondents said the sites are helpful and should be made more available while 26% considered them "a nuisance" that provide no benefit to patients who use them.

"Healthcare, like most all other industries, has clearly entered an era where measurements and reporting have increased importance," Peter Angood, MD, CEO of the ACPE, said in a statement. "This important new survey illustrates the strong concern among physician leaders about the quality and integrity of current reporting strategies and the data they are based upon."

The survey found that 69% of physicians had checked their online ratings. Of those respondents, 39% said they agreed with their ratings, while 19% didn't agree and 43% partially agreed with the findings. In a related question, some physicians were also skeptical of ratings compiled by organizations such as The Joint Commission and the National Committee for Quality Assurance. While most physicians (41%) said they were neutral about the rating systems, 29% said they were helpful and 14% described them as "a waste of time."

The study also included comments from respondents about the rating systems. One respondent said the ratings "are not very accurate" and that "the small sample can give a poor quality physician four stars and an excellent physician half a star." Another described the ratings as "a good idea in theory" but expressed frustration that "they allow angry people to do a lot of damage with no option for us to know what the issue is or how to address it."

Questionable questions
Some industry analysts say many online rating systems focus more on the patient's experience during an appointment than how it affected their overall health.

"Most of the rating systems are like customer satisfaction surveys," said Mark Anderson, a health information technology consultant and CEO of the AC Group in Texas. "You have some sites asking people about wait times and whether they were able to find a parking spot in the medical group's parking lot."

Part of the problem is that rating sites don't have access to clinical information such as patient outcomes that would help patients make more informed decisions.

"If I'm going to have knee surgery and I'm looking online for a surgeon, I don't want to know what their office looks like or how much time they spend with each patient," said Anderson. "I want to know if the surgeon has done five of these surgeries or 1,200 of them and the rate of complications patients experienced after their surgeries."

Insurer websites slightly better
Consumers interested in more detailed information about a physician have other options. Some major insurance companies have created physician rating sites that are more in line with the reporting systems favored by physicians, though they too have drawbacks.

"The concerns we had—though they are changing—were that the rating systems relied on an insufficient number of patients and did not include risk adjustments for the type of patients that physicians were treating," said Jeremy Lazarus, MD, president of the American Medical Association (AMA). "We were also concerned that some sites were focused more on cost of care rather than quality of care."

In a September 2010 letter sent to the nation's largest insurers, the AMA cited a Rand Corp. study that "called into question the use of cost-profiling tools" in rating physicians. But over the past few years, Lazarus said discussions between the AMA and insurers have resulted in a shift.

"Many insurers agreed with us and are now basing their ratings more on quality of care than on costs," he said.

Some disagreements between insurers and physicians over online ratings have resulted in legal action. In September 2010, the California Medical Association sought a restraining order against Blue Shield, requesting that the insurer take down its online Blue Ribbon Recognition Program.

The program evaluated physicians based on data from the California Physician Performance Initiative and its database with information on more than 5 million patients and 63,000 physicians. The lawsuit, which was eventually dismissed in 2011, claimed the rating program did not include information from patient medical charts, patient outcomes, or a record of previous patient treatments and relied on data based on five insurance plans sold in the state going back just three years.

The future
Provisions of the Patient Protection and Affordable Care Act could lead to the creation of new rating sites that use patient outcomes and clinical excellence as the basis for ratings.

"In theory, federal healthcare reform will require physicians to provide CMS with more data on quality of care," said Anderson. "But it's not clear how much of that data can be made available to patients."

Anderson said another outgrowth of healthcare reform—accountable care organizations—may provide a better vehicle for future rating systems, with hospitals, insurers, and medical groups sharing clinical results with the public to promote themselves as a destination for care.