Sorting the crawl
One product can sit under three category paths, each with its own address. A raw crawl lists those as three pages. Artificial intelligence (AI) sorts every crawled URL by template and flags patterns like that much faster than reading rows by hand. Which ones cost you traffic is a judgement call, and a person makes it, with your analytics open.


