Evolution of Scheduling

For this blog entry I have to thank Jonathan Mcalroy from Citi Group who wrote it. I have not modified it as I did not have anything to add. I think it is a very insightful look at scheduling, and instead of merely stating what is good or what is bad it focuses on what we should be doing and what we should be striving for. After all should we not be evolving instead of complaining, make it better or use it smarter is ultimately the better option.

I was asked recently about why we should move to r11 and what will be the long term benefits. It’s good question and I should have had a quick answer ready. I didn’t but fortunately for me it was an email exchange so I had time to properly formulate an answer. As it’s the anniversary of Darwin’s birthday, it’s timely and ironic that we can maintain our evolution with a little intelligent design.

For the reply, I could have rattled off a few paragraphs about the wonders of extended calendars and the marvels of look back dependencies. But I knew this questioner would push deeper and ask what benefits they would have. To answer the question therefore, I had to show the progression that all schedulers have been making in the last few decades so that I could then show the benefits that will follow from further advances.

An analogy I like to make is to imagine a line representing the business logic in a batch, if the green segment represents the logic held in the scheduler, then the blue portion is the logic held in the script executed by the scheduler. In the bad old days of Cron and Windows Task Scheduler, very little of the logic was in the scheduler. The vast majority of it was defined within the script that was getting executed.

As the early versions of AutoSys, TWS and Control-M came along, the amount of logic that could be included in the scheduler increased as it became possible to reuse the same job in different places and have conditions to define their execution.

As the schedulers have become more complex the number of job types has increased meaning that developers have had to do less developing in order to perform regular tasks. The 4.5 version of AutoSys has 3 (really just two) job types, whereas r11 has over a dozen and the ability to define new jobs. ITPAM has over a hundred possible ‘actions’ and new actions can be defined and made available to every other user.

The benefit of the migration of logic is that once it’s in a scheduler it’s very easy to re-use it in other places. Whether it’s a job that FTP’s a file or a job that executes a stored procedure, if it’s centrally managed it only requires definition once. This means that the pieces of logic they represent can be reused by multiple applications.

This means that instead of your batch just containing your logic, it can now contain lots of different groups logic in the right places. Your DB calls are managed by the DBA’s, the Grid calls are managed by the Grid team, the Business Objects feed is managed by the business objects team. Etc etc.

The ultimate goal is that a batch is reduced to a number of parameters that call pre-existing jobs to perform a required task.

In parallel to this line of evolution, the scalability of the scheduler has also had to evolve to keep up and handle all of these extra responsibilities. I remember using AutoSys 3.4 at Lehmans (RIP) and being shocked to hear that some sites ran two eventors in order to handle the workload. Fast forward a couple of years and the first time we ran 4.0 and wondered what EVENT_HDLR_ERROR meant.

In order to maintain evolution, the possibilities must always race ahead of the requirements. Every time we tell a developer, “No, AutoSys can’t do that” or “Hmm, you’ll have to script that” evolution is delayed. What we should be saying is; “Yes, the ABC team does that” or even, “Yes, here’s a wrapper script”. I’m not suggesting for a second that we start taking ownership of the business processes, but we should give our infrastructure colleagues the opportunity to get ahead of the curve and keep growing.

So in summary what does r11 give us? It might not seem like much but with a few new job types, the odd blob here and the WebService there, we can keep the line moving to the right and grow the expertise of our users and free up their resources.

“Ignorance more frequently begets confidence than does knowledge: it is those who know little, not those who know much, who so positively assert that this or that problem will never be solved by science.”

Charles Darwin

The Descent of Man

Tayori Limited