# All robots will spider the domain User-agent: * Disallow: # Disallow directory /images/ User-agent: * Disallow: /images/ Disallow: /media/ Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /wp-content/