Where the Reviewers Disagree, and by How Much

Update, 14 October 2026. We said both lower scorers of the Bose Smart Soundbar named the same two faults, thin bass and a narrow soundstage. Re-read on 14 October, the narrow soundstage is TechRadar’s complaint, made alongside the highest score; Tom’s Guide faulted inconsistent Dolby Atmos and bass without a subwoofer, and SoundGuys’ 6.8 is the average of five sub-scores that include a 1.0 for portability. The paragraph on the Bose is corrected. The totals below were not recomputed for the soundbar shortlist’s 14 October edition.

Update, 12 October 2026. Four shortlists were added, for phones, game consoles, security cameras and gaming laptops, so the totals are 248 scores across 71 products from 28 outlets, rather than 179 across 48 from 24. The median spread is unchanged at 1.0 point. Twenty-two products now span two points or more, not 20, and 16 of them sit at exactly 2.0, not 14: both new ones, the Nintendo Switch 2 and the Ring Pan-Tilt Indoor Cam, are single-step gaps, so the six spreads the star scale does not explain are the same six. Three outlets cleared the five-score bar for the first time: IGN at −0.45 from 7 scores, CNET at −0.32 from 7 and PCWorld at −0.01 from 6. IGN’s entry moves PCMag from second-harshest house to third. PCMag moved from −0.46 to −0.42, Tom’s Hardware from −0.21 to −0.26, Tom’s Guide from −0.18 to −0.09, Trusted Reviews from +0.27 to +0.25 and TechRadar from +0.31 to +0.35. Digital Trends (−0.50) and What Hi-Fi? (+0.44) did not move, so the gap between the harshest and kindest house is still 0.94 points; between PCMag and What Hi-Fi? it is 0.86 rather than 0.90.

Update, 8 October 2026. The portable speaker shortlist was re-read on 8 October. Two scores we had wrong were corrected: Tom’s Guide gives the JBL Flip 7 4.5 out of 5, not 4, and TechRadar gives the Sonos Roam 2 (affiliate link to Amazon, opens in a new tab) 4.5, not 4. Two speakers were added, so the totals are 179 scores across 48 products rather than 173 across 46. The median spread fell from 1.2 points to 1.0. Twenty products now span two points or more, not 21, and 14 of them sit at exactly 2.0, not 16: the Flip 7’s spread narrowed from 2.0 to 1.0, and the Roam 2’s widened from 2.0 to 3.0, which makes it a sixth spread the star scale does not explain. Tom’s Guide moved from −0.15 to −0.18, TechRadar from +0.28 to +0.31 and What Hi-Fi? from +0.49 to +0.44, so the gap between the harshest and kindest house is 0.94 points rather than 0.99, and between PCMag and What Hi-Fi? 0.90 rather than 0.95.

Update, 22 September 2026. The gaming handheld shortlist was re-read on 22 September and now carries 29 outlet scores rather than 10, so the totals here are 173 scores from 24 outlets rather than 154 from 22, still across 46 products. The median spread rose from 1.0 point to 1.2, and 21 products now span two points or more, 16 of them at exactly 2.0. The widest spread is no longer three points: the Legion Go S (SteamOS) runs from Trusted Reviews’ four and a half stars to The Verge’s 5 out of 10, four points apart. Digital Trends now has six scores, enough to count, and at −0.50 it is the harshest house; Tom’s Hardware enters the table at −0.21. PCMag moved from −0.55 to −0.46, Trusted Reviews from +0.23 to +0.27 and TechRadar from +0.31 to +0.28. The gap between the harshest and kindest house is still about 1.0 point, now measured from Digital Trends rather than PCMag.

Update, 21 September 2026. The robot vacuum and TV shortlists were re-read on 21 September, adding a net 11 outlet scores and four outlets we had not counted before, so the totals here are now 154 scores from 22 outlets rather than 143 from 18. Every per-outlet figure moved: PCMag from −0.60 to −0.55, Tom’s Guide from −0.24 to −0.15, Trusted Reviews from +0.18 to +0.23, TechRadar from +0.27 to +0.31 and What Hi-Fi? from +0.50 to +0.49. The gap between the harshest and kindest house narrowed from 1.1 points to 1.0, so we no longer say it has widened as the sample grew. The median spread, the twenty products spanning two points and the seventeen at exactly 2.0 are unchanged.

Reviewers agree less than an average makes them look. We hold 248 published scores across 71 products, and on a typical one the harshest and kindest outlet are a point apart on a ten-point scale. Twenty-two of the 71 span two points or more.

That matters because our shortlists print a single averaged number, and a single number hides whether it came from three people nodding along or two arguing. So here is the argument, from our own data.

The scale is doing a lot of the work

Before reading anything into those two-point gaps: most of them are one outlet giving four stars where another gave five. On a five-star scale, half a star is a whole point out of ten and one star is two. Sixteen of the twenty-two widest spreads land on exactly 2.0, which is the number a single step of disagreement produces.

That is a much smaller argument than “two points apart” sounds. It also means a five-star scale cannot express a small disagreement at all — the reviewer either matches or jumps a full point in our terms. Anyone averaging star ratings, us included, is working with a blunt instrument and should say so.

Six spreads survive that correction. The widest is the Legion Go S (SteamOS), at four points: Trusted Reviews’ four and a half stars against The Verge’s 5 out of 10, a score a star scale could not produce. The reviewers disagree about whether its Z2 Go chip is faster than a Steam Deck’s — a real argument about the machine, not the scale.

Two are ereaders. The Kobo Libra Colour and the Kindle Colorsoft each span a full three points, which takes a gap of one and a half stars rather than one — TechRadar’s five out of five against PCMag’s three and a half on the Kobo, and TechRadar’s four and a half against Trusted Reviews’ three on the Kindle. Both are arguments about whether the price is defensible rather than about how the device behaves. The Steam Deck OLED also spans three, from GamesRadar+’s full marks to Wired’s 7 out of 10, and so does the Sonos Roam 2: TechRadar’s four and a half stars against What Hi-Fi?’s three, the latter calling its sound unexceptional and its price a hike.

The Bose Smart Soundbar gets to 2.2 points another way: SoundGuys scored it 6.8 out of ten, a number a star scale cannot produce, against TechRadar’s 4.5 out of 5. SoundGuys’ number is the average of five sub-scores, one of them a 1.0 for portability, and Tom’s Guide’s three and a half stars came with praise for its music playback, so part of this gap is about what each outlet counts rather than how the bar sounds.

Outlets have house styles, and they are small

For every product with three or more scores, each outlet’s distance from the others reviewing the same thing was computed the same way. The pattern is consistent and modest.

Outlet	Scores	Distance from the room
Digital Trends	6	−0.50
IGN	7	−0.45
PCMag	26	−0.42
CNET	7	−0.32
Tom’s Hardware	10	−0.26
Tom’s Guide	59	−0.09
PCWorld	6	−0.01
Trusted Reviews	21	+0.25
TechRadar	55	+0.35
What Hi-Fi?	19	+0.44

Just under one point separates the harshest house from the kindest. That is larger than it looks: it is more than the difference between an 8.7 and a 9.3, which on one of our lists is the difference between second place and first.

Digital Trends is harshest at −0.50, but on six scores, which is barely more than the five-score floor; IGN, next at −0.45 on seven, and CNET and PCWorld, on seven and six, carry the same caution. The gap between PCMag, on 26 scores, and What Hi-Fi?, on 19, rests on far more scores than the Digital Trends figure, and it is 0.86 points. It widened as the sample first grew, then narrowed, from 1.1 points to 1.0 when PCMag’s scores for five more robot vacuums and TVs were counted, to 0.95 when the handheld scores were, to 0.90 when the speaker shortlist was re-read, and to 0.86 when four new shortlists added seven more PCMag scores.

What we are not claiming

Harsher is not better. An outlet sitting below the room may be tougher, or may review at a different price point, or may have had a worse unit. We cannot see which, and neither can you — a spread tells you an argument exists, not who won it.

The sample has grown with almost every update since this piece was first published, and the numbers moved: the median spread fell from 1.5 points to 1.0, then rose to 1.2 when one shortlist’s re-read added two wide spreads, fell back to 1.0 when another’s corrected two scores and added two products, and held at 1.0 when four new shortlists were added; the gap between the harshest and kindest house grew from 0.6 to 1.1, then eased to 1.0 and to 0.94, where the new shortlists left it. Small samples move. This one will move again.

The sample is also ours, which is a real limitation: these are products that made our shortlists, so they are mostly good hardware scoring in a narrow band near the top. A wider sample including bad products would probably show wider disagreement, not narrower.

How to read any shortlist, including ours

• Look for the spread, not just the average. An 8.5 from three outlets who all said 8.5 is a different fact from an 8.5 built out of a 7 and a 10.
• Treat a one-star gap as a small disagreement, because on a five-point scale it is the smallest one available.
• Take a score below 7 from an outlet that scores out of ten seriously. Those scales can express mild disappointment; star scales mostly cannot.
• Two agreeing outlets is a coincidence you can defend; three is a pattern. That is the threshold our shortlists use, and this data is the reason it is set where it is.

Every number here comes from scores we have already published with their sources attached, so you can check the arithmetic on any of them. That is the point of showing the working rather than the conclusion.