Mirakle Research · August 1, 2026 · revised August 6, 2026
The week our scraper thought Shopify deleted 148,172 reviews
For three days in July, Shopify's App Store stopped showing a large part of its reviews. Then it put most of them back. Our scraper watched the whole thing, wrote down every review that went, wrote down nothing when they returned, and reported a mass deletion that had not happened.
Three questions follow from that, and this report answers all three. Here they are up front, before any of the working.
Did reviews disappear?
Yes.
143,831 vanished from their listings over 16–18 July.
Did they come back?
Half.
73,635 returned. 70,196 never did.
Is the App Store smaller?
No.
It holds 770,302 reviews against 770,387 before — other reviews took their place.
That is the whole episode. The store's review count is where it was; a 4.8% slice of what you read today is reviews that were not visible a month ago, and roughly seventy thousand reviews that were visible are not.
This report replaces an earlier version. On 1 August 2026 we published this as 148,172 reviews Shopify had unpublished. Wrong twice over: that is a count of log entries, not of reviews. The correction we issued after that got the reason wrong as well. This is the third attempt, and this time each explanation has a query against it.
Why our number was 148,172
Our scraper measured the event correctly. On the peak day it saw 65,221 fewer reviews than two days earlier, and Shopify's counter said 64,633 fewer. It then did exactly what it is built to do: compared what it found against what it held, and wrote down every review it could no longer see.
It never wrote anything down when they came back. The scraper recorded a disappearance, then found the review again, silently un-marked it and logged nothing.
So a three-day outage entered the log as 148,172 departures and was never taken out.
And 148,172 is not even a count of reviews. It counts lines our log wrote. Reviews that went missing, came back and went missing again inside the same week wrote a line each time: 431 of them did, which is why the number of reviews involved is 147,738 and not 148,172. We published the larger figure and called it reviews.
Of those 147,738 reviews, 74,882 — 50.7% — are on their listings today. Tens of thousands were already back on the day we published.
The fix is one event type and it is now in place; returns are recorded from here on. It cannot be backfilled, because the information about when each review came back was never written down.
How many, and which ones
Two questions about a shelf of books look like the same question and are not. How many books are on it? And which books are on it? A shelf can hold exactly as many books as it did last month and not one of them the same.
That is the whole difficulty with this episode, and every number in this report answers one question or the other.
Which ones is our log. We identify a review and check whether it is still on its listing. Over the three days, 143,831 went missing — counted across all three days, so a review that went on the 16th and was back on the 18th is in that number.
How many is Shopify's count, printed on every listing. At the worst moment, on 17 July, the same 22,308 apps were 64,550 reviews short — the depth of the hole at one instant, not the traffic through it.
| Window | Removals we recorded | Our net | Shopify's net | Apart by |
|---|---|---|---|---|
| Announced sweep | 13,924 | −333 | +1,869 | 2,202 |
| 16–18 July | 93,534 | −64,503 | −11,352 | 53,151 |
| Fortnight after | 4,673 | +5,890 | +8,123 | 2,233 |
18 July is the day that settles it. Our log recorded 45,607 more reviews going missing. Shopify's count went up by 49,567.
Both are true. Reviews were coming back in enormous numbers that day and our scraper had no way to write that down — it recorded departures and was silent on returns. So the gross figure kept climbing while the shelf refilled underneath it.
What our own log recorded
We re-scan every app's review pages continuously and note when a review we had already recorded stops being there. In a normal week around 5,328 go that way. In the week of 13 July 2026, our log wrote 148,172 departure lines, across 6,978 apps. Shopify had announced a review-integrity sweep seven days earlier. There was a ready explanation sitting there and we took it.
Three days, and two counters moved at once
97.1% of the week's events fall on 16, 17 and 18 July — 143,831 reviews of the week's 147,738. The peak is 17 July, at 73,602. On the 19th it stops dead: 368, an ordinary day.
On the same three days a second counter moved. We record a review as newly discovered the first time we see it, and normally find about 1,185 a day. On 17 July we found 21,829. Disappearances and discoveries have no reason to move together.
We did not look at the second series at all the first time.
Four things that could have caused it
Three of them are our own equipment. Each one predicts something specific in the data. That is what makes them killable.
-
Ruled out
Our scraper was failing, or Shopify was throttling it, so it saw less than was there.
On the busiest day of the event our scraper made 113,178 requests. 6 failed and 1 was rate-limited — a failure rate of 0.0%, the same as every neighbouring day. Shopify answered essentially every request successfully. The pages it returned had fewer reviews on them.
-
Ruled out
Shopify re-issued review identifiers, so one review looked like a disappearance plus an arrival.
This was our first correction's explanation, and it does not survive. If identifiers had been rewritten, the reviews that vanished would reappear as the reviews that arrived. Matching the two sets on app, posted date, rating and review text finds 1,206 such pairs out of 143,831 disappearances — 0.8%. The identifiers themselves are also unchanged in format: the same numeric range before, during and after.
-
Ruled out
Our scraper stopped reading before the end of each app's reviews.
We used to hold a hard ceiling of 1,000 pages per app, which would have truncated exactly the biggest listings. That ceiling was removed on 28 May 2026, seven weeks before this, when we switched to paginating the newest-first review surface — so that particular failure was not available to us here. What remains is a per-app page budget of one page per ten reviews Shopify says the app has. It is sized to reach everything, but it takes its size from Shopify's count, and that count was falling. So it is worth testing, and it fails two ways. First, if we had stopped early, the pages we did fetch would still have been full: instead pages fetched fell 5.2% while reviews seen fell 8.2% — the pages we read were emptier, not fewer. Second, the budget rounds up, so an app with thirty reviews keeps the same three pages whether its count reads 30 or 27; a budget effect cannot touch small apps. Small apps lost the most: apps holding under fifty reviews lost 28.9% on average against 12.8% for apps holding five thousand or more.
-
This one survives
Shopify served fewer reviews than it had been serving.
We hold two independent measurements. One is what our scraper counted on the review pages. The other is the review total Shopify prints on each app's listing, which we read from a different page with different code and which involves no inference of ours. Across 10,016 apps scanned on both 15 and 17 July, our scraper saw 65,221 fewer reviews. Shopify's count fell by 64,633. The two disagree by 588, or 0.9%.
What Shopify's count did
Summing those printed counts is easy to get wrong. On a day our app scrape only reached part of the store the total drops by however much it missed, which looks exactly like a collapse. So the line below runs over a fixed panel: the 19,304 apps carrying a snapshot on every one of the 30 days we covered the whole store. Every point is the same apps. 4 days are missing from the range and the line breaks rather than drawing through them.
It falls on 16 July, bottoms out on the 17th at 8.9% below where it started, and by the 18th most of it is back. Then it climbs steadily for a fortnight. Whatever happened was not a deletion programme: deletions do not reverse themselves overnight.
How the shelf refilled
At the deepest point the same 22,308 apps were 64,550 reviews short. Most of that was back within a day. The rest came back over the following fortnight, and what remained is 2,569.
Set against a quiet fortnight before the event, which gained 1,132 reviews, the shortfall against trend is about 4,223.
The two measurements, over three windows
Both instruments measure the same apps, so they should agree: what arrived, minus what left, is how much the printed total moved. Across three windows they agree twice.
The first window is the sweep Shopify announced on 6 July. Our removals really were elevated there — 13,924 on the panel between 2 July and 15 July — and the two measurements land within 2,202 reviews of each other. A real removal programme reconciles. Across 16–18 July they are 53,151 apart, which is more than either of them reports as movement. In the fortnight after, they are back within 2,233.
| Window | Removals we recorded | Our net | Shopify's net | Apart by |
|---|---|---|---|---|
| Announced sweep | 13,924 | −333 | +1,869 | 2,202 |
| 16–18 July | 93,534 | −64,503 | −11,352 | 53,151 |
| Fortnight after | 4,673 | +5,890 | +8,123 | 2,233 |
One shopfront
One shopfront, day by day
Corpus numbers are hard to feel, so here is a small post-purchase upsell app — one listing among the six thousand — through the same three days. We are not naming it. Its size and category are what matter, and an app's review history is its shopfront.
It went into that week with 28 reviews. It came out with 42. Nothing about that is what you would predict from a story about reviews disappearing.
Thursday 16 July
Across the store, 24,722 reviews go missing and 7,448 appear. On this listing, 7 reviews vanish — and 8 arrive that had never been on the page as far as we could see, averaging 1,044 days old. Reviews from 2023, written by merchants who had long since moved on. The count goes from 28 to 34.
Friday 17 July
The worst day the store has: 73,602 reviews recorded gone, 21,829 appearing. On this listing, nothing leaves at all. Another 6 old reviews arrive, this batch averaging 852 days. The count reaches 42 — half again what it was on Tuesday.
Saturday 18 July
At 1:44 in the morning a Shopify staff member writes that some reviews were incorrectly unpublished and they are correcting it. Across the store our log records 45,607 more departures while Shopify's own total rises by nearly fifty thousand. On this listing the last movement of the week: 3 out, 3 in. It settles at 42 and has not moved since.
What the shopfront looks like now
17 reviews arrived across those three days and 16 are still on the listing. They average 922 days old and the oldest was written in November 2022. Meanwhile 5 of the app's own reviews left and never came back, and 4 went and returned.
So a merchant looking at this app today sees a longer review history than they would have seen a month ago, and roughly a third of it is testimony written years ago that was not on the page before. Nobody wrote a new word. The shelf refilled from somewhere the public could not see.
The reviews that arrived
Ordinary things, in the app's own language — its customers are largely French-speaking small merchants — thanking support staff by name.
The reviews that left and did not return
The reviews that left and came back
Read them side by side and there is nothing to separate the three groups. Same length, same warmth, same five stars, same named support agent. Whatever decided which pile each one landed in, it was not the review.
Chosen from more than a hundred candidates holding 18–60 reviews that lost some, recovered some, kept some off and had old reviews surface. Two apps with larger losses were rejected: their removed reviews were five-star, days-old and testimonial-shaped, naming the product in every sentence. We cannot show a review was solicited and will not imply it, so a case study resting on that suspicion is one we do not run.
Four groups of reviews
Those three days produced four populations, and they are not alike. 143,831 reviews left. 143,831 is that same set split two ways: the 73,635 Shopify put back and the 70,196 it did not. Alongside them, 42,099 reviews we had never recorded appeared.
Measured the same way against the live corpus, the differences are not subtle. What came back is the store: same average rating to three decimal places, same share of five stars, same share of complaints. What never came back is more critical and half again as long. What appeared is shorter, thinner and more glowing than either.
What came back
73,635 reviews, averaging 4.802 stars against the corpus's 4.804. Complaints run at 3.5% of them against 3.6% across the store.
There is nothing to say about this group except that it is ordinary, which is the whole point. Short, warm, mostly written in the first days of using something.
What never came back
70,196 reviews across 6,192 apps. Not a few listings taking a beating: the median affected app lost 3 reviews and the ten worst-hit hold 8.8% of the total.
This group is more critical than the store — 8.0% of it is one or two stars against 3.6% — and noticeably longer, a median 181 characters against 110.
But the sharpest difference is not in the reviews at all. It is in who wrote them.
For an account that had reviewed two to four apps, Shopify back roughly two thirds — 36.3% never came back. For an account that had reviewed 25+ apps, 98.6% never came back: of 3,902 removed reviews, 53 came back.
That is a clean gradient and we did not expect one. The rating did not decide it, the length did not decide it, the app did not decide it. The account did.
What we can say is that this is what Shopify's correction *did*, not why. Reviewing many apps is not evidence of anything by itself — agencies, consultants and merchants running several shops all do it, and the reviews in this group do not read as manufactured. They are specific. They name features. Several of them are complaints, which is not the usual shape of a purchased review.
We cannot see what Shopify saw. It has signals we do not — accounts, billing, install history, whatever its integrity checks actually read. A pattern that looks like a working merchant from the outside may look like something else from the inside. We can only report that the line it drew ran along the author, and that it drew it steeply.
What appeared instead
In the same three days, 42,099 reviews entered our records for the first time, average age 1,009 days. They are the most positive of the four groups — 92.3% five stars, 2.7% critical — the shortest, at a median 99 characters, and the most likely to carry no words at all (16.2%).
They are not the removed reviews under new identifiers; the twin test in section three rules that out. Nor are they a swap of the same people — only 2.5% of them were written by an account that had a review still missing.
Shopify's own documentation says reviews are published only when they meet its trust and quality standards, and that reviews move between published and archived states as merchant status changes. A pool of old, thin, five-star reviews sitting unpublished and then being published is consistent with that machinery running. Developers reported the same thing from their side of the listing, in the same days.
Every snippet on this page is hand-picked and hand-sanitised: no store, reviewer or support-agent name, no app named, and phrasing lightly changed where a sentence would be searchable back to its author. They illustrate; the numbers come from the queries.
What the developers saw, and what Shopify said
Everything above is one automated client's view of a website. That is the weakness of it.
The people whose listings these were noticed at the time, and wrote it down.
On 17 July — our trough — the Shopify developer forum filled up. At 6:13am one developer reported their count going “from 47 to 37 almost overnight”. At 7:21am another described losing “hundreds of reviews from real merchants”. At 9:09pm, in a thread titled “Review apocalypse”, a developer reported losing 60-odd reviews, 17% of their total.
At 1:44am on 18 July, a Shopify staff member replied in that thread that “some merchant reviews were incorrectly unpublished. We're working on correcting this now”. Later that morning a developer asked where to report reviews that had not come back yet, and another said most of theirs still had not.
Those are their words and their timestamps, not our measurement. Set them against the panel. It bottoms out on 17 July. Shopify says it is correcting the problem in the small hours of the 18th. The panel recovers most of the way that same day.
A developer looking at their own listing in a browser is not an automated client. They saw what we saw.
What we still cannot say
For three days Shopify served fewer reviews than it had the week before. Its own published counts agreed, to within one per cent, that the reviews were not there. Our equipment was not at fault. The developers affected saw it. Shopify called it a mistake and corrected it.
We cannot say what caused it. “Incorrectly unpublished” is Shopify's description of the outcome, not of the mechanism, and the review-integrity policy announced on 6 July describes a policy rather than an incident. Whether the rollout of that policy misfired, or something else did, is beyond what a scrape of the outside can establish.
We cannot cleanly separate what this episode removed from what the announced sweep removed in the same period, which is why the second of our two answers is a residual against a control rather than a count.
And one thing about our own reading deserves saying plainly. At the time, a review scan was treated as complete when it had fetched the number of pages Shopify's review count implied — it did not have to reach an empty page and prove it had seen everything. We changed that after this investigation: a scan now reads one page past the budget, and only a run that has seen an empty page is allowed to record a disappearance at all. The evidence above says stopping early does not explain the shape of what happened, but it is not something the July runs could have ruled out for themselves, and it would have inflated the count somewhat.
How this was measured
Two instruments. Ours: we scan every active app's review pages continuously, and a disappearance means a review we had already recorded on a listing and could not find on a later completed re-scan. We never infer one from a scan that failed or stopped early. Shopify's: the review count printed on each app listing, which we snapshot daily from a different page with different parsing code.
The panel is the 19,304 apps carrying a snapshot on every one of the 30 days between 2 July and 3 August on which our app scrape covered the whole store. Requiring every date costs about a fifth of the apps and removes all of the coverage noise: at 98% presence the sum still wobbles by more than the effect, because the missing 2% is not random and one absent app with ten thousand reviews moves the total on its own. The permanence measurement in section six uses a wider set — 22,308 apps present on the three anchor dates — and a 13-day quiet window as its control.
The baseline for our own disappearance counts is 4 May to 6 July 2026. Our first full pass, in late April, recorded every already-missing review at once; that week is excluded from every baseline figure and drawn in amber rather than dropped.
Blind spots, in order of how much they matter. We see only what Shopify served us, which is the limit described above. We only see reviews we captured in the first place, so our counts are lower bounds and an app's oldest reviews may be permanently outside our page depth. Shopify's printed count is itself a number Shopify computes; if it was lagging during those days, both our instruments would be wrong in the same direction and the loss would be understated. Reviews on apps that left the store entirely are outside the panel by construction. And a review can be removed and replaced inside a single scan interval without either instrument noticing.
Default store names. We identify a reviewer by store name and location, because that is all a listing shows, and Shopify gives every new shop a default name that plenty of merchants never change. “My Store, United States” therefore looks to us like one prolific reviewer with 710 apps to its name; across the placeholder names in every language we track, 237 of these fictional reviewers account for 8,815 reviews. Leave them in and the reach gradient becomes partly a chart about Shopify's onboarding. Every reviewer-level figure here excludes them.
Three totals, not one. Our database holds 1,034,720 review records, which is every review we have ever seen; removal in our data is a flag, not a deletion, so 161,585 of those are marked missing and still stored. The reviews we believe are actually live number 855,685. Shopify's count, summed on the last day of this study, comes to 832,838. A number that only ever goes up cannot answer a question about things going away.
Every figure on this page comes from one frozen snapshot, and you can read it yourself: the raw numbers behind this report — the same JSON the charts are drawn from, including the weekly series with our own backfill week flagged and the panel series with its gaps.
Revision record. First published 1 August 2026 as “The week Shopify unpublished 148,172 reviews”, reading our event count as a count of removals. Corrected 6 August 2026: the headline count is withdrawn, along with that version's findings about the age, length and authorship of what went — those described the reviews that were temporarily missing, not a set Shopify chose. An intermediate version of this correction attributed the three days to Shopify re-issuing review identifiers; the twin test in section three shows that was also wrong, and it has been replaced. The slug and URL are unchanged so existing links still work.
Sources, quoted as their claims and not our measurements. Shopify's changelog entry Strengthening trust in App Store reviews (6 July 2026) for the announced policy. For 16–18 July, the Shopify developer community threads “Review apocalypse: A perfect example of how to erode trust” (which carries the Shopify staff reply of 18 July, 1:44am), “Has anyone else noticed a sudden drop in their Shopify App Store reviews?”, “Has anyone experienced a sudden drop in Shopify App Store reviews?”, “Serious concerns about the recent app review changes”, “Destroying trust in App Store reviews” and “Shopify's new review policy is still unclear”. We found these after completing the measurement; the order matters, so it is stated here.