Indexing and Delisting
- robots.txt is not used to prevent a page from being indexed. Instead, a site owner can delist a page by using the robots noindex directive.
- Once your web page has been updated with this directive, Tavily needs to re-fetch it to apply the changes.