What is the best way to provide a search engine with crawl instructions?
Use structured data
Use a robots.txt file.
Use the search engine's Remove URL tool.
Use the tag.