Easily deploy a templated robots.txt file to all accounts within your WHM / cPanel server.

Ever need to quickly deploy a robots.txt template across all accounts on a WHM / cPanel server? The script below will allow you to do just that. It’s great for using in conjunction with a CRON job on your system to make sure all accounts are using the latest and greatest version of your robots file.

The Code

This can be handy when setting up a development / production / staging server. If used correctly, you can avoid having development links get picked up and indexed by google. This will also ensure that after a successful deployment to production, your site will not have issues being blocked from Google and other search engines.

After choosing a location to house the script, you can use the following crontab entry:

0 3 * * * /bin/bash /root/scripts/robotsDeploy.sh -f /root/scripts/blockRobots.txt >/dev/null 2>&1

My example is set to run at 3:00 am each morning. It will use the script found at /root/scripts/robotsDeploy.sh and utilize the template /roots/scripts/blockRobots.txt to automatically roll out that version of my robots.txt file to each and every cPanel and WHM account that exists on my server. You can easily change your template file, as well as what time your script runs by modifying the crontab entry.

What use cases have you come up with for this? In a future release I will setup a way to pass in a whitelist / blacklist of accounts which should be skipped automatically while inserting new rows into your robots.txt file.