Mirakle Research · September 4, 2026
Shopify is putting the July reviews back
Over 5 days from 3 September, 49,883 reviews returned to listings they had been missing from — about 9,977 a day against an ordinary 209. Then it stopped. 87.1% of them had disappeared during the same three days in July we wrote about twice, and of the 69,020 still missing when it started, 62.9% are back.
The first suspect
Whether we did this to ourselves
A number this size is our own instrument until we can show otherwise. Our log records a return when a scan finds a review we had marked as gone. Anything that made us look harder would produce the same shape: a deploy, a bigger crawl, a parser change.
Our last production release was 22 days earlier. On the busiest day of the returns our scraper made 111,654 requests and 25 of them failed, which is 0.02%. Neither number moved when the returns did.
The check that settles it is the second instrument. Shopify prints a review count on each app's listing page; we parse it with different code from a different page, and it has no way of knowing what our change log thinks. Summed over a panel of apps we read every single day, that count rose 4,189 on 3 September against an ordinary day's 62. Both instruments moved, on the same days, by comparable amounts. If this were ours, only one would have.
That is the licence for everything below. From here on, when the page says a review came back, it means Shopify's own listing says so too.
One listing, for scale
What this looked like from a single page
That licence is worth spending immediately, because the count Shopify prints is exactly what a developer sees when they open their own page. Every figure so far has been a corpus number, and a corpus number does not tell anyone what happened to them. So here is one listing — a gift-card app, mid-sized, chosen for the shape of its arc before any of its reviews were read. On 15 July its page showed 87 reviews at 4.3★.
The next morning seven were gone. The morning after that, fourteen more: 66 reviews left of 87, a fifth of the page missing in two days. Eight came back that week, and then nothing happened for a month and a half. From 18 July to 2 September the listing sat at 74 reviews and 4.5★ — 44 days at a number that was not right.
On 3 September they came back. The page now shows 89 reviews at 4.3★: two more than it started with, at exactly the rating it started with.
The rating is the part worth pausing on. For those 44 days the listing displayed 4.5★ — 0.2 of a star above what it shows now that its reviews are back. It was not being penalised. It was being flattered, because the review it lost first was the only critical one it had.
Works as advertised for the most part, but there is something every merchant should know before installing. In December our store was hit with a run of fraudulent gift-card orders and we were left to absorb the loss ourselves.
Really good product, and the support is the best part of it — quick to answer anything we asked. Very happy.
That first one is 1,090 characters from a merchant of two years standing, and it is a warning about losing money. It left on day one. It came back on 4 September, seven weeks later, among the last things to return. That is the rating gradient further down this page, happening to one page, to one developer, and to every shopper who read that listing in between.
The same fraud warning as above. It was the longest review on the listing and the only critical one, it left on the first day, and it was among the last things to come back.
Nobody told the developer any of this, and nothing on the page said it. Hold that listing in mind for the rest of the report: everything below is either that story multiplied by 6,946, or a reason to doubt that it generalises.
Ways to be wrong
The alternatives
One listing is a story and not an argument: a single page can move like that for reasons peculiar to itself. The corpus figures are what make it more than an anecdote, and those still leave several ways to be wrong. Each of the claims below would have made the finding smaller or different. Four died against a query. One is still partly open: we have rescanned 96.9% of the affected apps since the returns began, so a small tail of them has not been looked at since.
-
ruled out
Our code changed
Days between the last production deploy and the returns — 22 days earlier.
-
ruled out
Our scraper surged
Requests on the peak day, and the share that failed — 111,654 requests, 0.02% failed.
-
ruled out
We only looked harder
Shopify's own printed count on a panel read every day — rose 4,189 in a day.
-
ruled out
Shopify republished everything
Return rate of the removals Shopify announced on 6 July — 32.3% back, against 82.2%.
-
partly open
We have not finished looking
Share of affected apps rescanned since the returns began — 96.9% of 6,946 apps.
Three removals, three fates
The returns are aimed at one population
What survives those four is that Shopify put the reviews back. That is an attribution, not yet a finding: a company restoring everything it had lately removed would produce the same daily counts. So the question the agreement leaves us with is whether the returns distinguished between one removal and another. To answer that we need a removal we know was deliberate.
There is one. Shopify announced a review-integrity sweep on 6 July and carried it out over the following week: 20,980 removals made by the same company, on the same corpus, a fortnight before the ones we are measuring, and meant.
82.2% of the 16–18 July removals are back. 32.3% of the announced sweep is, which is 2.5 times less. Ordinary attrition from May and June sits between them at 45.1%. Whatever put these reviews back distinguished between a mistake and a decision, seven weeks after both had been made.
A selection implies something doing the selecting, and we cannot see it. What we can do is describe it the way you describe a tool you never saw: by the shape of the cut it left. The rest of this page is that shape.
The shape of the cut · first attempt
One-star reviews came back last, and least
The first thing to cut it by is the thing a review is mostly made of: a rating. We had already tried this once, and got it wrong. Our August report was called The praise came back. The complaints mostly didn't. When the returns began, 48.0% of the removals were still missing and the one-star reviews were missing at well above the five-star rate. That report measured a mid-point and read it as an endpoint, which is the hazard of writing about something while it is still moving. The direction held. The size did not.
Today 83.0% of the removed five-star reviews are back and 72.1% of the one-star ones, a gap of 10.9 points that runs cleanly down the scale with no step in it. The complaints did come back. They came back later, and fewer of them.
The absence of a step is the useful part. A rule that named a category, a developer or an app would show up here as a jump between two ratings and flatness elsewhere; a gradient that descends through all five is the signature of something ordered, applied to every review at once. But an ordering by rating is not the only ordering that would draw this line, and the nearest rival is size, because small apps and their ratings are not independent. So the next cut is by shelf.
The shape of the cut · the rival
The size gradient was inherited, not made
Apps holding fewer than ten reviews got back the equivalent of 9.03% of their whole shelf. Apps holding 5,000+ got 2.6%. The gradient runs straight down the size buckets, and that answers the rival: it is the same shape the removals had in July, at roughly the same steepness. The smallest listings lost the largest share of what they held, and they are getting the largest share back. The returns did not invent a size gradient. They reversed one, which is what an undoing looks like.
By category the spread is narrower. Product feeds leads at 8.83% of its shelf restored, across 101 apps. For an app with twenty reviews and four of them missing since July, this is not a rounding error in a corpus. It is most of a page.
Two gradients, then, and only one of them new. If the ordering by rating matters to anybody, it should be visible in the number a merchant actually reads.
The consequence
Almost nothing, once you take out the empty shelves
It is not visible there. The first cut says the average app gained +0.1109 stars across the returns. That number is an artifact and we nearly published it. 15,429 of the 26,167 apps we could compare held no reviews on 2 September, so their displayed rating was 0.0; give them one review back and they jump to five stars. They are three in five of the sample and they carry the whole effect.
Among apps that already had a review, the mean change is -0.0527 stars. Among apps with ten or more reviews that gained some, it is -0.003 — against +0.0003 in an ordinary two-day control window. Tens of thousands of reviews went back onto the shelf and the star averages did not move, because what came back is close enough to what was already there.
So the correction restored the count without disturbing the verdict. A merchant reading a rating in August and again in September saw the same number resting on a different, larger body of evidence. That is worth holding onto, because the count grew by more than the returns.
The other half of the arithmetic
Old reviews, arriving for the first time
Alongside the 49,883 reviews we watched come back, 21,040 more were recorded by our scraper for the first time, with no prior entry in our log at all. Their median age when we first saw them was 310 days. On an ordinary August day that figure is 2, and 80.5% of arrivals were posted within the fortnight; during the returns only 20.2% were.
These are reviews written years ago that were not on a listing when we scanned it and are on it now. The same signature appeared in July, which is the detail that makes this more than an oddity: the two events share a mechanism we cannot see but can recognise twice.
It also says something uncomfortable about the record. For as long as these reviews were absent, our corpus was wrong in a way no check inside it could have caught. The reviews were not missing from our copy of the listing; they were missing from the listing. An instrument cannot measure what it was never shown. We can count them; we cannot see where they were in between.
What survived
Where this leaves the two earlier reports
In July, 143,831 reviews left their listings over three days. 82.2% of them are back. The chain runs: two instruments agree, so it is Shopify's doing; the announced sweep stayed off, so it was selective; the selection descends the rating scale without a step, so it was ordered rather than categorical; and it moved no ratings, so what it corrected was the record and not the verdict.
The correction is aimed: the removals Shopify announced have stayed off at 32.3%, the ones nobody announced are back at 82.2%. Something distinguished between them seven weeks after the fact. Every link in that chain is a measurement. None of them is a motive, and we are not able to supply one: we have no statement from Shopify, and a public listing page does not record why anything on it is there.
The caveat is the tail. 96.9% of the 6,946 affected apps have been scanned since the returns began; the remaining 212 have not, so a small number of returns may still be unrecorded here. The rate has been back to normal for 7 days, which is what makes these figures worth quoting at all.
What we are left with is a boundary rather than a rule: the outline a decision leaves when you can see its results and not its reasons. The useful thing about a boundary is that it can be described precisely, and the 25,592 reviews on the wrong side of this one are described in the companion report. They are longer and more critical than the ones that returned, and they are not spread evenly across the store.
Method
How this was measured
Two instruments. Ours is the change log: a review we had recorded as gone from a listing which a later completed scan found on it again. Shopify's is the review count printed on each app's own listing page, parsed by different code from a different page, and summed each day over only those apps whose listing we read on every usable day of the window — a day counts as usable when it carries at least 60% of the median day's apps, which drops our own partial-scan days rather than letting them read as a collapse. The removals of 16-18 July are the population throughout; the removals Shopify announced on 6 July, and ordinary attrition from 3 May, are the controls. 1-2 May is excluded everywhere: that spike is our own first full rescrape recording every already-missing review at once. The returns are bounded by the daily rate rather than by assumption: they begin at 02:00 on 3 September, when restorations rise from 209 a day to roughly ten thousand, and end on 8 September, when the rate falls back to 242 a day. Every figure here is measured after they finished.
Definitions. A removal is a review we had recorded on a listing that a later completed scan did not find. A return is the reverse. We only began writing returns to the change log on 4 August; before that a review could come back without leaving a trace, which is why the population here is defined by the state a review is in today rather than by the absence of a log entry. The panel is every app whose listing we successfully read on every usable day between 1 July and the day before extraction — a day counts as usable when it carries at least 60% of the median day's apps. Apps that fall out of the panel do not move the line.
Limits. We cannot see Shopify's moderation decisions, only their effect on a public page. We cannot tell why a particular review came back or stayed off, and the star gradient is a pattern rather than a rule, since a monotonic gradient is consistent with several mechanisms and we are not choosing between them here. The rating comparison uses displayed ratings, which Shopify rounds.
Every number on this page comes from the snapshot behind it, which is published as raw JSON. The two earlier reports are the week of 13 July and what came back, and what didn't.