User-Agent: * Disallow: Crawl-delay: 1 User-Agent: Baiduspider Disallow: / User-Agent: Baiduspider-image Disallow: / User-Agent: Baiduspider-video Disallow: / User-Agent: Baiduspider-news Disallow: / User-Agent: Baiduspider-favo Disallow: / User-Agent: Baiduspider-cpro Disallow: / User-Agent: Baiduspider-ads Disallow: / User-Agent: DotBot Disallow: / User-Agent: spbot Disallow: / user-agent: AhrefsBot disallow: / User-agent: Bingbot Crawl-delay: 5 User-agent: MSNBot Crawl-delay: 5 User-agent: MSNBot-Media Crawl-delay: 5 User-agent: AdIdxBot Crawl-delay: 5 User-agent: BingPreview Crawl-delay: 5 Sitemap: https://internshala.com/sitemap.xml