By default, Screaming Frog will only crawl the subdomain you enter. Any additional subdomains that the spider encounters will be treated as external links. To crawl additional subdomains, you need to change the settings in the Spider Configuration menu. Checking Crawl All Subdomains will ensure that the SEO Spider crawls any links it encounters to other subdomains on your site. Step 1: how to crawl entire site – crawl all subdomains Step 2: If you start a scan from a specific subfolder or subdirectory and still want Screaming Frog to crawl the entire site, check the Crawl Outside of Start Folder box.
By default, SEO Spider is configured to only crawl the subfolder or subdirectory you are crawling.
If you want to crawl the entire site and italy phone number library start from a specific subdirectory. Make sure the configuration is set to crawl outside the starting folder. Tip : To save time and disk space, be mindful of resources you may not need during your crawl. Uncheck image, CSS, JavaScript, and SWF resources to reduce the crawl size. Want to get a comprehensive analysis of your website? Website address How to scan one subdirectory If you want to limit the scan to a single folder, simply enter the of the default settings.
If you overwrote the original default settings.
Reset the settings to default in the File menu. How to scan one subdirectory – default settings in file menu. If you want to start scanning in a specific folder, but need to continue scanning the rest of the subdomain, be sure to select Crawl Outside Of Start Folder in the Spider Configuration settings before entering your specific start URL. in spider configuration settings How to scan a specific set of subdomains in each of these sections you will be played a different audio recording or subdirectories To limit browsing to a specific set of subdomains or subdirectories, you can use Reg Ex to set these rules in the Include or Exclude options in the Configuration menu.
Exclusion In this example.
We looked at every page on elit-web.ru, excluding buy lead blog pages on each subdomain. Step 1 : Go to Configuration > Exclude, use wildcard regular expressions to define the URLs or parameters you want to exclude. configuration > exclude Step 2 : Test your regular expression to make sure it excludes the expected pages before you start crawling: check your regular expression Include In the example below, we wanted to view only the team subfolder on elit-web.ru. Again, use the Test tab to test multiple URLs and ensure RegEx is set up correctly for your inclusion rule.