Browse Source

Mise a jour des Disallow/Noindex pour permettres aux robots sociaux d'acceder aux images referencees dans les pages html, mais eviter aux robots d'indexation de parcourir ces memes dossiers. Au passage on bloque aussi les ?action= les *.api/ et la page de login

doctype_html5
Cerdic 2 months ago
committed by Gitea
parent
commit
7534b22f98
  1. 15
      robots.txt.html

15
robots.txt.html

@ -8,14 +8,17 @@
User-agent: *
Allow: /local/cache-css/
Allow: /local/cache-js/
Disallow: /local/
Noindex: /local/
Noindex: /plugins-dist/
Noindex: /plugins/
Noindex: /squelettes-dist/
Noindex: /squelettes/
Disallow: /ecrire/
Disallow: /plugins-dist/
Disallow: /lib/
Disallow: /plugins/
Disallow: /prive/
Disallow: /squelettes-dist/
Disallow: /squelettes/
Disallow: /spip.php?action=*
Disallow: /spip.php?page=login*
Disallow: /*.api/
Crawl-delay: 1
Sitemap: #URL_SITE_SPIP/sitemap.xml
Sitemap: #URL_SITE_SPIP/sitemap.xml
Loading…
Cancel
Save