Answers for "robots txt allow folder disallow subfolder"

0

robots txt allow folder disallow subfolder

User-agent: *
Allow: /news/$
Disallow: /news/
Explanation:

Google's robots.txt spec 
(https://developers.google.com/search/reference/robots_txt), 
which is more up to date than the "official" spec, states that:

/fish/ will match anything in the /fish/ folder but will not match 
/fish (and, no wildcard necessary, since "The trailing slash means 
this matches anything in this folder.") If you kinda reverse engineer that:

User-agent: * (or whatever user agent you want to talk to)
Allow: /news/$ (allows /news/ but the $ character says the allow 
can't go beyond /news/)
Disallow: /news/ (disallows anything in the /news/ folder)

Test it in Google Search Console, or in Yandex 
(https://webmaster.yandex.com/tools/robotstxt/) 
to ensure it works for your site.
Posted by: Guest on October-22-2020

Code answers related to "TypeScript"

Browse Popular Code Answers by Language