shuddr is a development framework which allows us access to the "deep Web" .
"Most of the Web's information is buried far down on dynamically generated sites, and standard search engines do not find it. Traditional search engines cannot "see" or retrieve content in the deep Web – those pages do not exist until they are created dynamically as the result of a specific search. The deep Web is several orders of magnitude larger than the surface Web." - Wikipedia.org
Among other things shuddr is a generic screen-scraping technology, but in a way which essentially turns websites into webservices. It's better referred to as a "directed web-crawler". shuddr as a framework helps you direct the crawling.