The archive · Developer & Business Tools · Product decision · 2020–2022
SearchMySite's anti-SEO-spam search bet: by May 2022 bots were ~100% of its traffic
A curated search engine for personal sites drew 0–13 real users a day while SEO bots fired ~163k searches — the 2022-05-16 HN thread drew 696 points
SearchMySite
What the business is
SearchMySite is a one-person, community-curated search engine that only indexes personal and independent (non-commercial) websites: every site must be submitted and verified by its owner, results carry no adverts, pages containing adverts are heavily downranked, and users are tracked with no persistent cookies. The plan was to fund running costs through search-as-a-service and listing fees rather than advertising or selling user data.
How it started
The operator built it as a solo side project in mid-2020 — first experiments at the end of May, initial code commit on 13 June, first site submissions on 17 July — after arguing that the ad-funded engines had been swamped by SEO spam and could no longer surface original non-commercial content. The launch post describes an Apache Solr index of verified personal and independent sites, deliberately run without advertising and with ad-carrying pages pushed down the rankings.
What happened
A retrospective published in January 2022 counted about 650 hours invested by the end of 2021, running costs near US$1,000 a year, seven paid listings worth roughly US$108 in annual revenue, and single-digit use of the paid features: nine users of custom exclusions, four on-demand indexing runs and one API-driven search box. Community moderation never scaled either — reviews of expired listings alone consumed whole weeks of the operator's time, and blog entries sometimes went a week with zero page views. Then, from mid-April 2022, automated search traffic exploded; the operator first suspected another engine scraping results, then a DDoS attack, before recognising 'scraping footprints' — queries built around 'Powered by' strings — typical of SEO tools that had listed searchmysite.net as a search engine.
How it ended up
Between 1 and 11 May 2022 the search page drew 0–13 real users a day (several of them the operator testing a release) while SEO spam bots fired 35,086–162,783 automated searches a day — 99.98–100% of all traffic, with 11 May at 100% spam and zero real users. The operator kept trying to block the flood, called it 'a losing battle, and one with nothing to gain by winning', and ended the post with 'this time I'm really not sure what the solution is'; the service was still live, and a 16 May update pointed readers to the Hacker News discussion.
Background
SearchMySite is a one-person, community-curated search engine that only indexes personal and independent (non-commercial) websites: every site must be submitted and verified by its owner, results carry no adverts, pages containing adverts are heavily downranked, and users are tracked with no persistent cookies. Its plan was to fund running costs through search-as-a-service and listing fees rather than advertising or selling user data.
The operator built it as a solo side project in mid-2020 — first experiments at the end of May, initial code commit on 13 June, first site submissions on 17 July — betting that people fed up with SEO-spam-filled results would search a smaller, curated index of original non-commercial content. A retrospective published in January 2022 counted about 650 hours invested, running costs near US$1,000 a year, seven paid listings worth roughly US$108, and single-digit use of the paid search-as-a-service features.
Then, from mid-April 2022, the picture collapsed into data: between 1 and 11 May the search page saw 0–13 real users a day (several of them the operator testing a release) while SEO spam bots fired 35,000–163,000 automated searches a day, 99.98–100% of all traffic. The queries carried 'scraping footprints' typical of SEO tools that had listed searchmysite.net as a search engine; the operator called it a likely losing battle and ended the post unsure the project could be saved. An update on 16 May pointed readers to the Hacker News discussion, which drew 696 points and 377 comments.
What has to be true
- Its entire pitch — original non-commercial content with no ads — made it valuable to black-hat operators, who found it through SEO tooling and mined it for fresh content and link targets.
- Discovery depended on viral links: the big engines sent almost no organic traffic, so a few HN visits were effectively the whole real audience and usage never compounded.
- The community-curation model assumed people would submit and moderate, but people did not arrive; reviews of expired listings alone consumed whole weeks of the solo operator's time.
- Revenue lagged the cost curve: roughly US$108 in paid listings against nearly US$1,000 a year in hosting, with search-as-a-service features drawing only a handful of users.
What can be applied
An anti-spam product advertises itself to spammers: SEO tools listed the engine and mined it for link-farm targets, while real users arrived only via HN links and bots outnumbered humans 30,000 to 1
Aftermath
As of 16 May 2022 SearchMySite was still live and the operator was fighting the flood: blocking automated SEO requests and weighing bot defences in GitHub issue #55, calling the fight 'a losing battle, and one with nothing to gain by winning'. The same-day update pointed readers to the Hacker News discussion (696 points, 377 comments), where commenters debated bot-blocking and the Dead Internet Theory. Still ahead was the January 2022 plan to spend less time on the project, cut blogging and move to cheaper hosting around mid-2022.
Sources
- Almost all searches on my independent search engine are now from SEO spam bots
- Almost all searches on my independent search engine are now from SEO spam bots
spotted an error? The archive wants to know.
Your turn
You just read one. Describe what you are building, and see who is betting on the same thing.
Free account · 3 free questions · no card