diff options
| author | Rafi Zadanly <zadanlyr@gmail.com> | 2023-05-16 11:22:00 +0700 |
|---|---|---|
| committer | Rafi Zadanly <zadanlyr@gmail.com> | 2023-05-16 11:22:00 +0700 |
| commit | 711993092eb75aacc9766f0da1c264e5dde55218 (patch) | |
| tree | cd04ce332260829f3e516dd589ba742d7d966e04 | |
| parent | 63f200e5d142dce9155a34fe02d3a58161fc6762 (diff) | |
add robots.txt
| -rw-r--r-- | public/robots.txt | 13 |
1 files changed, 10 insertions, 3 deletions
diff --git a/public/robots.txt b/public/robots.txt index 62114b1f..a4ee103d 100644 --- a/public/robots.txt +++ b/public/robots.txt @@ -1,10 +1,17 @@ # This says to apply these settings to ALL search engine spiders/crawlers # User-agent: * + +User-agent: AhrefsBot +User-agent: Adsbot-Google +User-agent: Yandex User-agent: Googlebot -Disalow: /shop/product/pdf/* -Disalow: /shop/product/https://* +Disallow: /shop/product/pdf/* +Disallow: /shop/product/https://* Disallow: /v1/* -Disallow: /sentral.indoteknik.com* + +User-agent: * +Disallow: sentral.indoteknik.com/* +Disallow: erp.indoteknik.com/* Sitemap: https://indoteknik.com/sitemap/products.xml Sitemap: https://indoteknik.com/sitemap/brands.xml |