Blogs

The Heritrix crawler is capable of crawling and capturing web content fairly reliably from blogs built upon popular template-based platforms like Wordpress. Problems specific to capturing blogs typically manifest when those blogs are incorporated into the structure of a larger web instance with its own domain. Recognizing links to these incorporated blogs and their contents as in fact paths to domains external to the main seed’s, the crawler will sometimes erroneously deem these URLs outside of the scope of the given collection.

To ensure that a broader web presence’s blog and its embedded content capture completely, follow the directions for expanding the a collection’s scope to include a specific host, making sure to include both the blog’s and any associated file-hosting subdomain’s URLs.