Running Python Cron Jobs in Windows Azure
I have some python scripts that store some data in my SQL Azure database and I want them run daily from my web role. Why my web role? Because I only have a web tier and it’s not worth dedicating a worker for this task. It took about 5 hours today getting this just right, so here is what I did in case anyone else needs to do something similar. First, thanks to Steve Marx for some very helpful posts on the topic of Azure startup task.
First, I included the python installer and pyodbc.pyd (for the SQL connection) in my web role project, as well as my python scripts themselves. I marked the scripts as Content so that I can reference them in the approot directory. I also marked the python installer and pyodbc as “Copy to Output directory” : “Copy always” so I can run them from my bat file on startup.
I then create the installcrawler.cmd bat file that contains the following:
1: install python 2.7
2: copy pyodbc to the python directory
3: start the task scheduler - this is not started by default on Azure
4-7: setup my scheduled tasks to run when needed. Note that in my case, the timezone on the VM is GMT
Then kick-off a deployment, get a cup of coffee, write a blog post, and then RDP into the VM to make sure all is well. If you need a refresher course on “at” and scheduled tasks as I did, here is a good reference.
UPDATE 10/13: As Steve Marx pointed out when he read this post, this will only work on a single instance web role. If we add more instances, then the tasks will run in each instance, which in my case is not what I want to happen. So, here is a fix to allow for multi-instance:
Rather than running installcrawler.cmd (shown above) at startup, run a powershell file that only installs the python scripts if we are on the first instance. See that below.
Again, thanks to Steve Marx for some great blog posts on the topics of running powershell scripts at startup. Note that the $role.Id.Contains(“_0”) test is what limits the installation to the first instance. Obviously, if this instance is down for the monthly OS upgrade (or anything else), your scripts won’t run. This is fine for us, as this is a background task that doesn’t need to happen exactly on schedule 100% of the time.
