The Open Guide to Glasgow

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.

===== Scraping: DNS Filter =====

This code takes the unique URLs (see [[Scrape/Hidden Glasgow]] for an example and checks which ones are pointing to domains that no longer exists (then drops them from the to-be-checked list).

<code php dns-filter.php>
<?php
$file_list = "non-image-links.txt";

$data = file_get_contents($file_list);
$lines = explode("\n", $data);
$host_list = array();

foreach($lines as $url) {
        $url_parts = parse_url($url);
        if(!empty($url_parts['host'])) {
                $host = $url_parts['host'];
                $host_list[$host][] = $url;
        }
}

foreach($host_list as $host_name=>$url_list) {
        if(checkdnsrr($host_name, "A")) {
                foreach($url_list as $url) {
                        echo $url."\n";
                }
        } else {
                // you can echo out the failing domains here
                // echo "ERROR: $host_name\n";
        }
}

?>
</code>

I run this to dump the valid links to a new file:

<code bash>
php dns-filter.php > filtered-url-list.txt
</code>

The Open Guide to Glasgow

User Tools

Site Tools

Page Tools