档案库 · 开发与企业工具 · 产品决策 · 2020–2022
SearchMySite 反SEO垃圾信息的搜索赌注：到2022年5月，机器人占其流量的约100%
一个面向个人网站的精选搜索引擎每天只有0-13个真实用户，而SEO机器人每天发起约16.3万次搜索——2022年5月16日的HN帖子获得696分
SearchMySite
做的是什么生意
SearchMySite is a one-person, community-curated search engine that only indexes personal and independent (non-commercial) websites: every site must be submitted and verified by its owner, results carry no adverts, pages containing adverts are heavily downranked, and users are tracked with no persistent cookies. The plan was to fund running costs through search-as-a-service and listing fees rather than advertising or selling user data.
起因
The operator built it as a solo side project in mid-2020 — first experiments at the end of May, initial code commit on 13 June, first site submissions on 17 July — after arguing that the ad-funded engines had been swamped by SEO spam and could no longer surface original non-commercial content. The launch post describes an Apache Solr index of verified personal and independent sites, deliberately run without advertising and with ad-carrying pages pushed down the rankings.
经过
A retrospective published in January 2022 counted about 650 hours invested by the end of 2021, running costs near US$1,000 a year, seven paid listings worth roughly US$108 in annual revenue, and single-digit use of the paid features: nine users of custom exclusions, four on-demand indexing runs and one API-driven search box. Community moderation never scaled either — reviews of expired listings alone consumed whole weeks of the operator's time, and blog entries sometimes went a week with zero page views. Then, from mid-April 2022, automated search traffic exploded; the operator first suspected another engine scraping results, then a DDoS attack, before recognising 'scraping footprints' — queries built around 'Powered by' strings — typical of SEO tools that had listed searchmysite.net as a search engine.
结果
Between 1 and 11 May 2022 the search page drew 0–13 real users a day (several of them the operator testing a release) while SEO spam bots fired 35,086–162,783 automated searches a day — 99.98–100% of all traffic, with 11 May at 100% spam and zero real users. The operator kept trying to block the flood, called it 'a losing battle, and one with nothing to gain by winning', and ended the post with 'this time I'm really not sure what the solution is'; the service was still live, and a 16 May update pointed readers to the Hacker News discussion.
背景
SearchMySite是一个单人运营、社区精选的搜索引擎，只索引个人和独立（非商业）网站：每个网站必须由所有者提交并验证，结果不含广告，含广告页面被大幅降权，用户跟踪不使用持久cookies。其计划是通过搜索即服务和列表费来覆盖运营成本，而非广告或出售用户数据。
运营者于2020年年中将其作为个人副业项目构建——5月底开始实验，6月13日首次代码提交，7月17日首批网站提交——押注厌倦SEO垃圾结果的用户会搜索更小、精选的原创非商业内容索引。2022年1月发布的回顾统计了约650小时的投入，年运营成本近1000美元，7个付费列表约108美元，付费搜索即服务功能使用次数为个位数。
然后，从2022年4月中旬起，情况在数据中崩溃：5月1日至11日，搜索页每日真实用户为0-13人（其中几个是运营者在测试发布），而SEO垃圾机器人每天发起35,000至163,000次自动化搜索，占总流量的99.98%至100%。查询带有“抓取足迹”，是SEO工具的典型特征，这些工具已将searchmysite.net列为搜索引擎；运营者称这可能是一场必败之战，并在帖子结尾表示不确定项目能否保住。5月16日的更新将读者指向Hacker News上的讨论，该讨论获得696分和377条评论。
这件事要成立，得有什么
- 其整个卖点——无广告的原创非商业内容——使其对黑帽运营者很有价值，他们通过SEO工具发现它，并挖掘新鲜内容和链接目标。
- 流量发现依赖病毒式链接：大型引擎几乎没有带来有机流量，因此HN的少量访问实际上就是全部真实受众，使用量从未复合增长。
- 社区精选模式假设人们会提交和审核，但人们没有来；仅过期列表的审核就消耗了独立运营者数周时间。
- 收入滞后于成本曲线：付费列表约108美元，而托管每年近1000美元，搜索即服务功能只有少数用户。
可借鉴之处
反垃圾产品向垃圾制造者自我宣传：SEO工具将该引擎列入列表并挖掘链接农场目标，而真实用户仅通过HN链接到达，机器人数量是人类的30000比1
后续进展
截至2022年5月16日，SearchMySite仍然在线，运营者正在与洪水作斗争：在GitHub问题#55中阻止自动SEO请求并权衡机器人防御措施，称这场战斗为“必败之战，赢了也无利可图”。同日更新将读者指向Hacker News上的讨论（696分，377条评论），评论者讨论了机器人阻止和死互联网理论。展望未来，还有2022年1月的计划：减少项目时间，减少博客，并在2022年年中左右转向更便宜的托管。
资料来源
- Almost all searches on my independent search engine are now from SEO spam bots
- Almost all searches on my independent search engine are now from SEO spam bots
发现哪里写错了？告诉我们。
轮到你了
你刚读完一家。说说你在做什么，看看谁在赌同一件事。
免费账号 · 3 次免费提问 · 不用绑卡