# ════════════════════════════════════════════════════════════════════ # robots.txt # Policy: fully open to traditional search engines AND AI platforms on # public marketing content; thin/machine endpoints excluded to keep the # primary index clean. Transactional, private, and Free Preview AJAX # surfaces are disallowed. # # What IS crawlable (every bot): # • / home, courses catalogue, course detail # • /Home/CourseDetails/... buy page (the conversion target — IS indexed) # • /podcasts, /glossaries, /guides, /stats, /tables marketing hubs # • /career-academy SEO hub # • /sitemap.xml # # What is NOT crawlable (every bot): # • /Admin/, /AdminDashboard/, /Account/, /Dashboard/ private/transactional # • /Support/, /Payment/, /B2B/, /Employee/, /Messenger/ private # • /Quiz/ preview content + paid-user surface # • /Home/UnitDetails/ duplicate of CourseDetails (canonical there) # • /Home/LoadContentStream LLM-backed SSE endpoint (machine-only) # • /Home/GetUnitVideos LLM/SerpAPI-backed AJAX (machine-only) # • /Home/GetUnitNews LLM/SerpAPI-backed AJAX (machine-only) # • /Home/GetUnitScholar LLM/SerpAPI-backed AJAX (machine-only) # • /Home/GenerateUnitsBackground LLM-backed AJAX (machine-only) # ════════════════════════════════════════════════════════════════════ # ─── General crawlers (Googlebot, Bingbot, DuckDuckBot, Yandex, etc.) ─── User-agent: * Allow: / Allow: /podcasts Allow: /podcasts/ Allow: /podcasts/rss.xml Allow: /podcasts/llms.txt Allow: /glossaries Allow: /glossaries/ Allow: /glossaries/rss.xml Allow: /glossaries/llms.txt Allow: /guides Allow: /guides/ Allow: /guides/rss.xml Allow: /guides/llms.txt Allow: /stats Allow: /stats/ Allow: /stats/rss.xml Allow: /stats/llms.txt Allow: /tables Allow: /tables/ Allow: /tables/rss.xml Allow: /tables/llms.txt Allow: /diagrams Allow: /diagrams/ Allow: /diagrams/rss.xml Allow: /diagrams/llms.txt Allow: /library Allow: /library/ Allow: /library/rss.xml Allow: /library/llms.txt Allow: /videos Allow: /videos/ Allow: /videos/rss.xml Allow: /videos/llms.txt Allow: /career-academy Allow: /career-academy/ Allow: /career-academy/llms.txt Allow: /news Allow: /news/ Allow: /careers Allow: /careers/ # Thin / machine-only endpoints kept out of the index Disallow: /podcasts/search? Disallow: /podcasts/Feed Disallow: /glossaries/search? Disallow: /guides/search? Disallow: /stats/search? Disallow: /tables/search? Disallow: /diagrams/search? Disallow: /library/search? Disallow: /videos/search? Disallow: /news/bookmarks Disallow: /news/following Disallow: /careers/bookmarks Disallow: /careers/alerts # Transactional, private and duplicate surfaces Disallow: /Admin/ Disallow: /AdminDashboard/ Disallow: /Account/ Disallow: /Dashboard/ Disallow: /Support/ Disallow: /Payment/ Disallow: /Quiz/ Disallow: /Employee/ Disallow: /B2B/ Disallow: /ExternalApi/ Disallow: /Messenger/ # ─── Bonus Pillar (2026-05-08) — Free Preview surfaces are NOT for crawlers ─── # Unit content is duplicate of CourseDetails (which IS crawled and ranks). The # AJAX endpoints below burn LLM credits per first-time hit; they are machine-only, # not for indexing. The rest of the public marketing site stays fully crawlable. Disallow: /Home/UnitDetails/ Disallow: /Home/LoadContentStream Disallow: /Home/GetUnitVideos Disallow: /Home/GetUnitNews Disallow: /Home/GetUnitScholar Disallow: /Home/GenerateUnitsBackground # ─── Crawl-budget control (2026-09-11, after the 10-day crawl storm) ───────── # The same course is reachable under {city}/{country}/ prefixes and /lang/ and # /variant/ suffixes — an effectively infinite URL space. Crawlers took the # pools down enumerating it (Bingbot/Baiduspider full renders at 100+ KB each). # Only the canonical /Home/CourseDetails/{id}/{title} form is crawlable. Disallow: /*/Home/ Disallow: /Home/CourseDetails/*/lang/ Disallow: /Home/CourseDetails/*/variant/ Disallow: /*searchTerm= Disallow: /*?city= Disallow: /*&city= # Bing/Yandex/Baidu honor Crawl-delay (Google ignores it and adapts on its own). # 10s = max ~6 pages/min per crawler — full renders are expensive (LLM/DB). Crawl-delay: 10 Sitemap: https://learnuni.com/sitemap.xml # ─── AI platforms: explicitly allowed on all public content ───────── # (Each bot gets its own group so they cannot fall through to the # general wildcard group if they ever ignore it. Every group repeats # the private + preview Disallow rules so an AI bot following its own # group cannot reach the LLM-backed endpoints.) User-agent: GPTBot Allow: / Disallow: /podcasts/search? Disallow: /podcasts/Feed Disallow: /glossaries/search? Disallow: /guides/search? Disallow: /stats/search? Disallow: /tables/search? Disallow: /Admin/ Disallow: /AdminDashboard/ Disallow: /Account/ Disallow: /Dashboard/ Disallow: /Support/ Disallow: /Payment/ Disallow: /Quiz/ Disallow: /Employee/ Disallow: /B2B/ Disallow: /Messenger/ Disallow: /Home/UnitDetails/ Disallow: /Home/LoadContentStream Disallow: /Home/GetUnitVideos Disallow: /Home/GetUnitNews Disallow: /Home/GetUnitScholar Disallow: /Home/GenerateUnitsBackground Disallow: /ExternalApi/ Disallow: /*/Home/ Disallow: /Home/CourseDetails/*/lang/ Disallow: /Home/CourseDetails/*/variant/ Disallow: /*searchTerm= Disallow: /*?city= Disallow: /*&city= User-agent: ChatGPT-User Allow: / Disallow: /podcasts/Feed Disallow: /Admin/ Disallow: /Account/ Disallow: /Dashboard/ Disallow: /Payment/ Disallow: /Quiz/ Disallow: /B2B/ Disallow: /Messenger/ Disallow: /Home/UnitDetails/ Disallow: /Home/LoadContentStream Disallow: /Home/GetUnitVideos Disallow: /Home/GetUnitNews Disallow: /Home/GetUnitScholar Disallow: /Home/GenerateUnitsBackground Disallow: /ExternalApi/ Disallow: /*/Home/ Disallow: /Home/CourseDetails/*/lang/ Disallow: /Home/CourseDetails/*/variant/ Disallow: /*searchTerm= Disallow: /*?city= Disallow: /*&city= User-agent: OAI-SearchBot Allow: / Disallow: /podcasts/search? Disallow: /podcasts/Feed Disallow: /Admin/ Disallow: /Account/ Disallow: /Dashboard/ Disallow: /Payment/ Disallow: /Quiz/ Disallow: /B2B/ Disallow: /Messenger/ Disallow: /Home/UnitDetails/ Disallow: /Home/LoadContentStream Disallow: /Home/GetUnitVideos Disallow: /Home/GetUnitNews Disallow: /Home/GetUnitScholar Disallow: /Home/GenerateUnitsBackground Disallow: /ExternalApi/ Disallow: /*/Home/ Disallow: /Home/CourseDetails/*/lang/ Disallow: /Home/CourseDetails/*/variant/ Disallow: /*searchTerm= Disallow: /*?city= Disallow: /*&city= User-agent: PerplexityBot Allow: / Disallow: /podcasts/search? Disallow: /podcasts/Feed Disallow: /Admin/ Disallow: /Account/ Disallow: /Dashboard/ Disallow: /Payment/ Disallow: /Quiz/ Disallow: /B2B/ Disallow: /Messenger/ Disallow: /Home/UnitDetails/ Disallow: /Home/LoadContentStream Disallow: /Home/GetUnitVideos Disallow: /Home/GetUnitNews Disallow: /Home/GetUnitScholar Disallow: /Home/GenerateUnitsBackground Disallow: /ExternalApi/ Disallow: /*/Home/ Disallow: /Home/CourseDetails/*/lang/ Disallow: /Home/CourseDetails/*/variant/ Disallow: /*searchTerm= Disallow: /*?city= Disallow: /*&city= User-agent: Perplexity-User Allow: / Disallow: /Admin/ Disallow: /Account/ Disallow: /Dashboard/ Disallow: /Payment/ Disallow: /Quiz/ Disallow: /B2B/ Disallow: /Messenger/ Disallow: /Home/UnitDetails/ Disallow: /Home/LoadContentStream Disallow: /Home/GetUnitVideos Disallow: /Home/GetUnitNews Disallow: /Home/GetUnitScholar Disallow: /Home/GenerateUnitsBackground Disallow: /ExternalApi/ Disallow: /*/Home/ Disallow: /Home/CourseDetails/*/lang/ Disallow: /Home/CourseDetails/*/variant/ Disallow: /*searchTerm= Disallow: /*?city= Disallow: /*&city= User-agent: ClaudeBot Allow: / Disallow: /podcasts/search? Disallow: /podcasts/Feed Disallow: /Admin/ Disallow: /Account/ Disallow: /Dashboard/ Disallow: /Payment/ Disallow: /Quiz/ Disallow: /B2B/ Disallow: /Messenger/ Disallow: /Home/UnitDetails/ Disallow: /Home/LoadContentStream Disallow: /Home/GetUnitVideos Disallow: /Home/GetUnitNews Disallow: /Home/GetUnitScholar Disallow: /Home/GenerateUnitsBackground Disallow: /ExternalApi/ Disallow: /*/Home/ Disallow: /Home/CourseDetails/*/lang/ Disallow: /Home/CourseDetails/*/variant/ Disallow: /*searchTerm= Disallow: /*?city= Disallow: /*&city= User-agent: Claude-Web Allow: / Disallow: /Admin/ Disallow: /Account/ Disallow: /Dashboard/ Disallow: /Payment/ Disallow: /Quiz/ Disallow: /B2B/ Disallow: /Messenger/ Disallow: /Home/UnitDetails/ Disallow: /Home/LoadContentStream Disallow: /Home/GetUnitVideos Disallow: /Home/GetUnitNews Disallow: /Home/GetUnitScholar Disallow: /Home/GenerateUnitsBackground Disallow: /ExternalApi/ Disallow: /*/Home/ Disallow: /Home/CourseDetails/*/lang/ Disallow: /Home/CourseDetails/*/variant/ Disallow: /*searchTerm= Disallow: /*?city= Disallow: /*&city= User-agent: anthropic-ai Allow: / Disallow: /Admin/ Disallow: /Account/ Disallow: /Dashboard/ Disallow: /Payment/ Disallow: /Quiz/ Disallow: /B2B/ Disallow: /Messenger/ Disallow: /Home/UnitDetails/ Disallow: /Home/LoadContentStream Disallow: /Home/GetUnitVideos Disallow: /Home/GetUnitNews Disallow: /Home/GetUnitScholar Disallow: /Home/GenerateUnitsBackground Disallow: /ExternalApi/ Disallow: /*/Home/ Disallow: /Home/CourseDetails/*/lang/ Disallow: /Home/CourseDetails/*/variant/ Disallow: /*searchTerm= Disallow: /*?city= Disallow: /*&city= User-agent: Google-Extended Allow: / Disallow: /podcasts/search? Disallow: /podcasts/Feed Disallow: /Admin/ Disallow: /Account/ Disallow: /Dashboard/ Disallow: /Payment/ Disallow: /Quiz/ Disallow: /B2B/ Disallow: /Messenger/ Disallow: /Home/UnitDetails/ Disallow: /Home/LoadContentStream Disallow: /Home/GetUnitVideos Disallow: /Home/GetUnitNews Disallow: /Home/GetUnitScholar Disallow: /Home/GenerateUnitsBackground Disallow: /ExternalApi/ Disallow: /*/Home/ Disallow: /Home/CourseDetails/*/lang/ Disallow: /Home/CourseDetails/*/variant/ Disallow: /*searchTerm= Disallow: /*?city= Disallow: /*&city= # 2026-09-12: this group used to be "Allow: /podcasts" and nothing else. # A crawler obeys only the most specific group that names it, and that group # replaces the wildcard group rather than adding to it - so Googlebot-News was # the one crawler permitted to walk /{city}/{country}/Home/..., the ?city= # duplicates, /ExternalApi/ and the rest. Cloudflare skips verified Google # crawlers past the geo block and the rate limit, so it had an unmetered path # to the most expensive endpoints here, and it used it: LCFT and LearnUNI both # went down with those URLs filling the request queue. User-agent: Googlebot-News Allow: / Allow: /news Allow: /news/ Allow: /podcasts Allow: /podcasts/ Disallow: /podcasts/search? Disallow: /podcasts/Feed Disallow: /glossaries/search? Disallow: /guides/search? Disallow: /stats/search? Disallow: /tables/search? Disallow: /diagrams/search? Disallow: /library/search? Disallow: /videos/search? Disallow: /news/bookmarks Disallow: /news/following Disallow: /Admin/ Disallow: /AdminDashboard/ Disallow: /Account/ Disallow: /Dashboard/ Disallow: /Support/ Disallow: /Payment/ Disallow: /Quiz/ Disallow: /Employee/ Disallow: /B2B/ Disallow: /Messenger/ Disallow: /ExternalApi/ Disallow: /Home/UnitDetails/ Disallow: /Home/LoadContentStream Disallow: /Home/GetUnitVideos Disallow: /Home/GetUnitNews Disallow: /Home/GetUnitScholar Disallow: /Home/GenerateUnitsBackground Disallow: /*/Home/ Disallow: /Home/CourseDetails/*/lang/ Disallow: /Home/CourseDetails/*/variant/ Disallow: /*searchTerm= Disallow: /*?city= Disallow: /*&city= User-agent: CCBot Allow: / Disallow: /podcasts/search? Disallow: /podcasts/Feed Disallow: /Admin/ Disallow: /Account/ Disallow: /Dashboard/ Disallow: /Payment/ Disallow: /Quiz/ Disallow: /B2B/ Disallow: /Messenger/ Disallow: /Home/UnitDetails/ Disallow: /Home/LoadContentStream Disallow: /Home/GetUnitVideos Disallow: /Home/GetUnitNews Disallow: /Home/GetUnitScholar Disallow: /Home/GenerateUnitsBackground Disallow: /ExternalApi/ Disallow: /*/Home/ Disallow: /Home/CourseDetails/*/lang/ Disallow: /Home/CourseDetails/*/variant/ Disallow: /*searchTerm= Disallow: /*?city= Disallow: /*&city= User-agent: FacebookBot Allow: / Disallow: /Admin/ Disallow: /Account/ Disallow: /Dashboard/ Disallow: /Payment/ Disallow: /Quiz/ Disallow: /B2B/ Disallow: /Messenger/ Disallow: /Home/UnitDetails/ Disallow: /Home/LoadContentStream Disallow: /Home/GetUnitVideos Disallow: /Home/GetUnitNews Disallow: /Home/GetUnitScholar Disallow: /Home/GenerateUnitsBackground Disallow: /ExternalApi/ Disallow: /*/Home/ Disallow: /Home/CourseDetails/*/lang/ Disallow: /Home/CourseDetails/*/variant/ Disallow: /*searchTerm= Disallow: /*?city= Disallow: /*&city= User-agent: facebookexternalhit Allow: / Disallow: /Admin/ Disallow: /Account/ Disallow: /Dashboard/ Disallow: /Payment/ Disallow: /Quiz/ Disallow: /B2B/ Disallow: /Messenger/ Disallow: /Home/UnitDetails/ Disallow: /Home/LoadContentStream Disallow: /Home/GetUnitVideos Disallow: /Home/GetUnitNews Disallow: /Home/GetUnitScholar Disallow: /Home/GenerateUnitsBackground Disallow: /ExternalApi/ Disallow: /*/Home/ Disallow: /Home/CourseDetails/*/lang/ Disallow: /Home/CourseDetails/*/variant/ Disallow: /*searchTerm= Disallow: /*?city= Disallow: /*&city= User-agent: meta-externalagent Allow: / Disallow: /Admin/ Disallow: /Account/ Disallow: /Dashboard/ Disallow: /Payment/ Disallow: /Quiz/ Disallow: /B2B/ Disallow: /Messenger/ Disallow: /Home/UnitDetails/ Disallow: /Home/LoadContentStream Disallow: /Home/GetUnitVideos Disallow: /Home/GetUnitNews Disallow: /Home/GetUnitScholar Disallow: /Home/GenerateUnitsBackground Disallow: /ExternalApi/ Disallow: /*/Home/ Disallow: /Home/CourseDetails/*/lang/ Disallow: /Home/CourseDetails/*/variant/ Disallow: /*searchTerm= Disallow: /*?city= Disallow: /*&city= User-agent: Amazonbot Allow: / Disallow: /podcasts/search? Disallow: /podcasts/Feed Disallow: /Admin/ Disallow: /Account/ Disallow: /Dashboard/ Disallow: /Payment/ Disallow: /Quiz/ Disallow: /B2B/ Disallow: /Messenger/ Disallow: /Home/UnitDetails/ Disallow: /Home/LoadContentStream Disallow: /Home/GetUnitVideos Disallow: /Home/GetUnitNews Disallow: /Home/GetUnitScholar Disallow: /Home/GenerateUnitsBackground Disallow: /ExternalApi/ Disallow: /*/Home/ Disallow: /Home/CourseDetails/*/lang/ Disallow: /Home/CourseDetails/*/variant/ Disallow: /*searchTerm= Disallow: /*?city= Disallow: /*&city= User-agent: Applebot Allow: / Disallow: /Admin/ Disallow: /Account/ Disallow: /Dashboard/ Disallow: /Payment/ Disallow: /Quiz/ Disallow: /B2B/ Disallow: /Messenger/ Disallow: /Home/UnitDetails/ Disallow: /Home/LoadContentStream Disallow: /Home/GetUnitVideos Disallow: /Home/GetUnitNews Disallow: /Home/GetUnitScholar Disallow: /Home/GenerateUnitsBackground Disallow: /ExternalApi/ Disallow: /*/Home/ Disallow: /Home/CourseDetails/*/lang/ Disallow: /Home/CourseDetails/*/variant/ Disallow: /*searchTerm= Disallow: /*?city= Disallow: /*&city= User-agent: Applebot-Extended Allow: / Disallow: /Admin/ Disallow: /Account/ Disallow: /Dashboard/ Disallow: /Payment/ Disallow: /Quiz/ Disallow: /B2B/ Disallow: /Messenger/ Disallow: /Home/UnitDetails/ Disallow: /Home/LoadContentStream Disallow: /Home/GetUnitVideos Disallow: /Home/GetUnitNews Disallow: /Home/GetUnitScholar Disallow: /Home/GenerateUnitsBackground Disallow: /ExternalApi/ Disallow: /*/Home/ Disallow: /Home/CourseDetails/*/lang/ Disallow: /Home/CourseDetails/*/variant/ Disallow: /*searchTerm= Disallow: /*?city= Disallow: /*&city= User-agent: Bytespider Allow: / Disallow: /podcasts/search? Disallow: /podcasts/Feed Disallow: /Admin/ Disallow: /Account/ Disallow: /Dashboard/ Disallow: /Payment/ Disallow: /Quiz/ Disallow: /B2B/ Disallow: /Messenger/ Disallow: /Home/UnitDetails/ Disallow: /Home/LoadContentStream Disallow: /Home/GetUnitVideos Disallow: /Home/GetUnitNews Disallow: /Home/GetUnitScholar Disallow: /Home/GenerateUnitsBackground Disallow: /ExternalApi/ Disallow: /*/Home/ Disallow: /Home/CourseDetails/*/lang/ Disallow: /Home/CourseDetails/*/variant/ Disallow: /*searchTerm= Disallow: /*?city= Disallow: /*&city= User-agent: DuckAssistBot Allow: / Disallow: /Admin/ Disallow: /Account/ Disallow: /Dashboard/ Disallow: /Payment/ Disallow: /Quiz/ Disallow: /B2B/ Disallow: /Messenger/ Disallow: /Home/UnitDetails/ Disallow: /Home/LoadContentStream Disallow: /Home/GetUnitVideos Disallow: /Home/GetUnitNews Disallow: /Home/GetUnitScholar Disallow: /Home/GenerateUnitsBackground Disallow: /ExternalApi/ Disallow: /*/Home/ Disallow: /Home/CourseDetails/*/lang/ Disallow: /Home/CourseDetails/*/variant/ Disallow: /*searchTerm= Disallow: /*?city= Disallow: /*&city= User-agent: MistralAI-User Allow: / Disallow: /Admin/ Disallow: /Account/ Disallow: /Dashboard/ Disallow: /Payment/ Disallow: /Quiz/ Disallow: /B2B/ Disallow: /Messenger/ Disallow: /Home/UnitDetails/ Disallow: /Home/LoadContentStream Disallow: /Home/GetUnitVideos Disallow: /Home/GetUnitNews Disallow: /Home/GetUnitScholar Disallow: /Home/GenerateUnitsBackground Disallow: /ExternalApi/ Disallow: /*/Home/ Disallow: /Home/CourseDetails/*/lang/ Disallow: /Home/CourseDetails/*/variant/ Disallow: /*searchTerm= Disallow: /*?city= Disallow: /*&city= User-agent: cohere-ai Allow: / Disallow: /Admin/ Disallow: /Account/ Disallow: /Dashboard/ Disallow: /Payment/ Disallow: /Quiz/ Disallow: /B2B/ Disallow: /Messenger/ Disallow: /Home/UnitDetails/ Disallow: /Home/LoadContentStream Disallow: /Home/GetUnitVideos Disallow: /Home/GetUnitNews Disallow: /Home/GetUnitScholar Disallow: /Home/GenerateUnitsBackground Disallow: /ExternalApi/ Disallow: /*/Home/ Disallow: /Home/CourseDetails/*/lang/ Disallow: /Home/CourseDetails/*/variant/ Disallow: /*searchTerm= Disallow: /*?city= Disallow: /*&city= User-agent: Diffbot Allow: / Disallow: /Admin/ Disallow: /Account/ Disallow: /Dashboard/ Disallow: /Payment/ Disallow: /Quiz/ Disallow: /B2B/ Disallow: /Messenger/ Disallow: /Home/UnitDetails/ Disallow: /Home/LoadContentStream Disallow: /Home/GetUnitVideos Disallow: /Home/GetUnitNews Disallow: /Home/GetUnitScholar Disallow: /Home/GenerateUnitsBackground Disallow: /ExternalApi/ Disallow: /*/Home/ Disallow: /Home/CourseDetails/*/lang/ Disallow: /Home/CourseDetails/*/variant/ Disallow: /*searchTerm= Disallow: /*?city= Disallow: /*&city= User-agent: YouBot Allow: / Disallow: /Admin/ Disallow: /Account/ Disallow: /Dashboard/ Disallow: /Payment/ Disallow: /Quiz/ Disallow: /B2B/ Disallow: /Messenger/ Disallow: /Home/UnitDetails/ Disallow: /Home/LoadContentStream Disallow: /Home/GetUnitVideos Disallow: /Home/GetUnitNews Disallow: /Home/GetUnitScholar Disallow: /Home/GenerateUnitsBackground Disallow: /ExternalApi/ Disallow: /*/Home/ Disallow: /Home/CourseDetails/*/lang/ Disallow: /Home/CourseDetails/*/variant/ Disallow: /*searchTerm= Disallow: /*?city= Disallow: /*&city= User-agent: PetalBot Allow: / Disallow: /Admin/ Disallow: /Account/ Disallow: /Dashboard/ Disallow: /Payment/ Disallow: /Quiz/ Disallow: /B2B/ Disallow: /Messenger/ Disallow: /Home/UnitDetails/ Disallow: /Home/LoadContentStream Disallow: /Home/GetUnitVideos Disallow: /Home/GetUnitNews Disallow: /Home/GetUnitScholar Disallow: /Home/GenerateUnitsBackground Disallow: /ExternalApi/ Disallow: /*/Home/ Disallow: /Home/CourseDetails/*/lang/ Disallow: /Home/CourseDetails/*/variant/ Disallow: /*searchTerm= Disallow: /*?city= Disallow: /*&city= Crawl-delay: 10