User-agent: Scrapy Allow: / User-Agent: * Allow: /wp-content/uploads/ Disallow: /wp-content/plugins/ Disallow: /wp-admin/ Disallow: /wp-content/uploads/2019/08/SCOR-Shopping-Centres-2018_19.pdf Disallow: /wp-content/uploads/2019/12/SCOR-Report_Offices-2019_web.pdf