Crawling is the traversal of pages by search engine bots; indexing is their recording in the index so they can be displayed. The robots.txt file controls crawling, the meta robots tag (noindex) controls indexing, and an XML sitemap helps discover URLs. A URL blocked by robots.txt can still appear in the index.
Example
<meta name="robots" content="noindex">