# Meshini Trips - Tourist UI # https://www.robotstxt.org/ # ============================================ # ALLOWED - Legitimate Search Engines # ============================================ # Google (all variants) User-agent: Googlebot User-agent: Googlebot-Image User-agent: Googlebot-Mobile User-agent: Googlebot-News User-agent: Googlebot-Video User-agent: AdsBot-Google User-agent: AdsBot-Google-Mobile User-agent: Mediapartners-Google User-agent: Google-InspectionTool Allow: / # Microsoft Bing (also powers Yahoo, Ecosia, Qwant) User-agent: Bingbot User-agent: msnbot User-agent: msnbot-media User-agent: adidxbot Allow: / # DuckDuckGo User-agent: DuckDuckBot User-agent: DuckDuckGo-Favicons-Bot Allow: / # Baidu (China) User-agent: Baiduspider User-agent: Baiduspider-image User-agent: Baiduspider-video Allow: / # Yandex (Russia) User-agent: YandexBot User-agent: YandexImages User-agent: YandexVideo User-agent: YandexMedia Allow: / # Naver (Korea) User-agent: NaverBot User-agent: Yeti Allow: / # Seznam (Czech Republic) User-agent: SeznamBot Allow: / # Sogou (China) User-agent: Sogou User-agent: sogou spider Allow: / # Shenma (China Mobile) User-agent: YisouSpider Allow: / # Brave Search User-agent: Bravebot Allow: / # Perplexity AI (allowed AI search) User-agent: PerplexityBot Allow: / # WolframAlpha User-agent: WolframAlphaBot Allow: / # Boardreader User-agent: BoardReader Allow: / # Mojeek (independent UK search) User-agent: MojeekBot Allow: / # Qwant (French search) User-agent: Qwantify Allow: / # Social Media Crawlers (for link previews) User-agent: facebookexternalhit User-agent: Facebot User-agent: Twitterbot User-agent: LinkedInBot User-agent: WhatsApp User-agent: TelegramBot User-agent: Slackbot User-agent: Slackbot-LinkExpanding User-agent: Applebot User-agent: Pinterest User-agent: Pinterestbot User-agent: Discordbot User-agent: SkypeUriPreview User-agent: vkShare User-agent: Embedly Allow: / # Archive.org (web archive) User-agent: ia_archiver Allow: / # ============================================ # BLOCKED - SEO/Marketing Bots # ============================================ User-agent: AhrefsBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: DotBot Disallow: / User-agent: BLEXBot Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: PetalBot Disallow: / User-agent: SearchmetricsBot Disallow: / User-agent: SEOkicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: sistrix Disallow: / User-agent: SISTRIX Crawler Disallow: / User-agent: Screaming Frog SEO Spider Disallow: / User-agent: MegaIndex Disallow: / User-agent: Majestic Disallow: / User-agent: Rogerbot Disallow: / User-agent: Exabot Disallow: / User-agent: Uptimebot Disallow: / User-agent: LinkpadBot Disallow: / User-agent: SEOPowerSuite Disallow: / User-agent: SiteExplorer Disallow: / # ============================================ # ALLOWED - AI Search/Chat (cites sources) # ============================================ User-agent: ChatGPT-User Allow: / # ============================================ # BLOCKED - AI Training Crawlers # ============================================ User-agent: CCBot Disallow: / User-agent: GPTBot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Claude-Web Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Amazonbot Disallow: / User-agent: Bytespider Disallow: / User-agent: cohere-ai Disallow: / User-agent: meta-externalagent Disallow: / User-agent: FacebookBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Diffbot Disallow: / User-agent: Omgilibot Disallow: / User-agent: YouBot Disallow: / # ============================================ # BLOCKED - Scrapers & Malicious Bots # ============================================ User-agent: Scrapy Disallow: / User-agent: Python-urllib Disallow: / User-agent: python-requests Disallow: / User-agent: HTTrack Disallow: / User-agent: WebCopier Disallow: / User-agent: WebZIP Disallow: / User-agent: Offline Explorer Disallow: / User-agent: SiteSnagger Disallow: / User-agent: TeleportPro Disallow: / User-agent: WebStripper Disallow: / User-agent: wget Disallow: / User-agent: curl Disallow: / User-agent: libwww-perl Disallow: / User-agent: Go-http-client Disallow: / User-agent: Java Disallow: / User-agent: HttpClient Disallow: / User-agent: Nutch Disallow: / User-agent: Heritrix Disallow: / User-agent: archive.org_bot Disallow: / User-agent: MauiBot Disallow: / User-agent: Cliqzbot Disallow: / User-agent: Sogou inst spider Disallow: / User-agent: AspiegelBot Disallow: / # ============================================ # DEFAULT - Block sensitive paths for all # ============================================ User-agent: * Disallow: /trap/ Disallow: /callback Disallow: /silent-renew Disallow: /api/ # Crawl-delay for unknown bots Crawl-delay: 10 # Sitemap Sitemap: https://meshini.com/sitemap.xml