Where our data comes from

Four kinds of source, each good for a different thing, each with a bias worth stating. Plus what we cannot reach, and what we refuse to use.

bitcritiq has not used any of the products it writes about. Everything here is built from other people’s work, and this page says whose, how it is gathered, and what each source is and is not good for. The specific evidence for any individual claim is cited on the article that makes it — this page is the method, not the evidence.

The four corpora

1. Published outlet reviews — for scores

The only thing we average. A product appears on a shortlist when three named outlets have published a numeric score for it; fewer than three is a citation, not a consensus. Every score is shown with its original scale, its conversion to ten, and a link to the review.

Most-used, by volume of scores currently on the site: Tom’s Guide, TechRadar, What Hi-Fi?, PCMag, Trusted Reviews and Digital Trends, with SoundGuys, Tom’s Hardware, Homes & Gardens, Expert Reviews, TechGearLab, Notebookcheck, PCWorld, PC Gamer, CNET, WIRED, ITPro and Engadget appearing where they have reviewed a product the others also covered.

The bias: outlets review what manufacturers send them, which favours brands that ship few, well-covered models and penalises those shipping many variants. That is why Garmin is missing from our smartwatch shortlist and no single Samsung soundbar qualifies — a fact about product strategy, not product quality.

2. Community aggregators — for leads only, never for findings

Reddit’s own search is blocked to us. Aggregators that poll it are therefore how we find out a discussion exists at all — RedditRecs is the one we currently use, covering 50-plus categories and updating weekly.

We treat these as a pointer, never as a source of record, and the reason is specific: it publishes no methodology. No ranking algorithm, no per-category data volumes, no filtering or de-duplication rules, no attribution for the individual reports it aggregates. Its sentiment percentages are therefore unauditable, and where we quote one we attribute it to them rather than adopting it as ours.

The working pattern is: the aggregator says a conversation exists, we go and read it, and anything factual gets checked against a primary source before publication.

3. Owner forums — for the faults reviews cannot see

A review runs for days or a fortnight. The things that ruin a product — brushes that clog, mop pads that sour, a dock that jams, an app that degrades, a maker that stops shipping firmware — arrive over months. Forums are the only corpus that sees them.

The bias is severe and worth stating every time: people post when something breaks and stay silent when it works, enthusiast forums are not a cross-section of owners, and vote counts measure entertainment as much as usefulness. The top-voted posts in the largest robot vacuum community this year are jokes and photographs of cats riding the machines.

Consequently forum material never touches a score, and an owner-signal block only appears where at least twenty reports have been read and each theme carries a counted share. Where a corpus ranks themes without counting them, we report the themes in prose and publish no block — a drawn bar is a measurement claim.

4. Primary sources — for anything factual

Manufacturer specification and pricing pages, regulator filings, company announcements and financial disclosures. Anything a reader could act on gets traced back to the organisation that said it.

This is the one that has repeatedly changed a story rather than decorating it. A community claim that the US had banned Roombas turned out to be lawmakers requesting a national security review. A claim that the FCC had banned Roborock vacuums turned out to be a Covered List update defining the class as “mobile robots, such as humanoids and quadrupeds”, which does not name vacuums and does not affect models already authorised. Both threads were pointing at something real that no review covered. Neither said what it was reported to say.

What we cannot reach

Stated because it shapes what we can write, not to complain about it.

  • Reddit search is blocked to us. Listings and individual threads are not, so we can read a community but not search it.
  • RTINGS keeps its picks behind a paywall. We exclude it rather than cite what we cannot read.
  • Several major outlets — among them Reuters, AP, The Verge and Ars Technica — block our fetcher entirely. Where a story depends on them we route through a browser or use a different outlet, and if we cannot verify it we do not run it.
  • Search volume. Our topic selection uses Google Trends, which gives relative interest and rising terms, not absolute volume, difficulty or commercial value. A term described here as a strong signal is fast-rising, which is not the same as large.

What we refuse to use

Amazon customer reviews. Two reasons, and the second matters more than the first. Collecting them at volume is against Amazon’s terms and would put our Associates account at risk. And they are a weak corpus regardless: heavily gamed, subject to review-hijacking across listing variants, anonymous, and clustered in the first fortnight of ownership — exactly the window professional reviews already cover, and months away from where the real faults appear.

Anything we cannot link to. If a number cannot be traced to a page a reader can open, it does not go on the site.

Our testing standard covers what would have to be true before anything here carries a score of ours, and the editorial policy covers who writes what.