diff options
| author | IT Fixcomart <it@fixcomart.co.id> | 2023-05-17 01:55:14 +0000 |
|---|---|---|
| committer | IT Fixcomart <it@fixcomart.co.id> | 2023-05-17 01:55:14 +0000 |
| commit | e16d278e6c63177d488bb8b5c62cedc898906c09 (patch) | |
| tree | 2b2a20ea2ac439ec2945f921f21f9160cd7b6c28 /public/robots.txt | |
| parent | 6608d0e62fd232cbc19f42f58d1f56cfe2ef6842 (diff) | |
| parent | 0abaa462194715c25cee5fcede0da3085939b402 (diff) | |
Merged in master (pull request #8)
Master
Diffstat (limited to 'public/robots.txt')
| -rw-r--r-- | public/robots.txt | 19 |
1 files changed, 19 insertions, 0 deletions
diff --git a/public/robots.txt b/public/robots.txt new file mode 100644 index 00000000..a4ee103d --- /dev/null +++ b/public/robots.txt @@ -0,0 +1,19 @@ +# This says to apply these settings to ALL search engine spiders/crawlers +# User-agent: * + +User-agent: AhrefsBot +User-agent: Adsbot-Google +User-agent: Yandex +User-agent: Googlebot +Disallow: /shop/product/pdf/* +Disallow: /shop/product/https://* +Disallow: /v1/* + +User-agent: * +Disallow: sentral.indoteknik.com/* +Disallow: erp.indoteknik.com/* + +Sitemap: https://indoteknik.com/sitemap/products.xml +Sitemap: https://indoteknik.com/sitemap/brands.xml +Sitemap: https://indoteknik.com/sitemap/categories.xml +Sitemap: https://indoteknik.com/sitemap/videos.xml
\ No newline at end of file |