diff options
| author | Rafi Zadanly <zadanlyr@gmail.com> | 2023-05-12 17:01:37 +0700 |
|---|---|---|
| committer | Rafi Zadanly <zadanlyr@gmail.com> | 2023-05-12 17:01:37 +0700 |
| commit | 344fe1857b8307933082110c15b737af9118ac89 (patch) | |
| tree | 2451a7988dd1427d292afd23872320f30270de9a /public/robots.txt | |
| parent | 961d68ecdd7cb503889fc23e47490673d52bb5ac (diff) | |
add robots.txt
Diffstat (limited to 'public/robots.txt')
| -rw-r--r-- | public/robots.txt | 12 |
1 files changed, 12 insertions, 0 deletions
diff --git a/public/robots.txt b/public/robots.txt new file mode 100644 index 00000000..62114b1f --- /dev/null +++ b/public/robots.txt @@ -0,0 +1,12 @@ +# This says to apply these settings to ALL search engine spiders/crawlers +# User-agent: * +User-agent: Googlebot +Disalow: /shop/product/pdf/* +Disalow: /shop/product/https://* +Disallow: /v1/* +Disallow: /sentral.indoteknik.com* + +Sitemap: https://indoteknik.com/sitemap/products.xml +Sitemap: https://indoteknik.com/sitemap/brands.xml +Sitemap: https://indoteknik.com/sitemap/categories.xml +Sitemap: https://indoteknik.com/sitemap/videos.xml
\ No newline at end of file |