User-agent: * # Content Signals (contentsignals.org / draft-romm-aipref-contentsignals). # Westview's position: maximum visibility. AI assistants may retrieve this # content to answer questions, may use it as model input, and may train on it. Content-Signal: ai-train=yes, search=yes, ai-input=yes Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: /wp-content/uploads/wc-logs/ Disallow: /wp-content/uploads/woocommerce_transient_files/ Disallow: /wp-content/uploads/woocommerce_uploads/ # Cart, checkout, add-to-cart and wc-ajax URLs are NOT disallowed here on purpose. # They were (until 2026-10-12), and every one that had already been crawled sat in # Search Console as "Indexed, though blocked by robots.txt": a blocked URL can never # be re-fetched, so Google never sees that it should drop. Each one now says so # itself: cart/checkout carry a noindex meta, and westview_noindex_action_urls / # WooCommerce send "X-Robots-Tag: noindex" for ?add-to-cart= and wc-ajax. Do not re-add. # Feeds are NOT disallowed here on purpose. Blocking them (2026-09-08) left # already-indexed feeds stuck as "Indexed, though blocked by robots.txt", # because Google could never re-fetch them to see they should drop. The theme # now sends "X-Robots-Tag: noindex" on every feed (westview_noindex_feeds) and # no longer links per-page feeds, so let crawlers reach them. Do not re-add. # WooCommerce Store API. The read-only product catalog is deliberately open — # it is what /.well-known/api-catalog advertises to agents, so blocking it here # would tell well-behaved agents not to fetch the API we just pointed them at. # Cart/checkout/order/batch stay closed: session-stateful and useless to a # crawler. Longest-match wins, so these Disallows beat the Allow above them. # Everything else under /wp-json/wc/ (notably the wc/v3 admin API) stays blocked. Allow: /wp-json/wc/store/v1 Disallow: /wp-json/wc/store/v1/cart Disallow: /wp-json/wc/store/v1/checkout Disallow: /wp-json/wc/store/v1/order Disallow: /wp-json/wc/store/v1/batch Disallow: /wp-json/wc/ Disallow: /wp-*.php Disallow: /product-tag/*/page/ # Spurious video-embed URLs used to be Disallow-ed here. That kept Googlebot # from fetching them, which meant it could never see they were dead: two sat in # Search Console as "Indexed, though blocked by robots.txt" and the rest as # "Blocked by robots.txt" forever. They now return a hard 410 from the theme # (westview_gone_410_defunct_pages), so letting crawlers reach them is what # actually gets them dropped. Do not re-add these Disallows. Sitemap: https://www.westviewmfg.com/sitemap.xml Sitemap: https://www.westviewmfg.com/image-sitemap.xml Sitemap: https://www.westviewmfg.com/ca/sitemap.xml Sitemap: https://www.westviewmfg.com/ca/image-sitemap.xml