What is the best way to provide a search engine with crawl instructions?
Use the tag.
Use structured data
Use the search engine's Remove URL tool.
Use a robots.txt file.