Workflows and batch files
-
Out of curiosity, do any of you (in a professional capacity, as in, the company you work for) have various automated jobs that run at night the still execute batch files? And said jobs are put into a workflow with predecessor / successor dependencies, such that if one of the .bat processes (which can be, say, an EXE or a SQL SP call) fails, the whole dependency chain fails? Or does your company use a more sophisticated way of dealing with automating workflows and dependencies?
Latest Article - Building a Prototype Web-Based Diagramming Tool with SVG and Javascript Learning to code with python is like learning to swim with those little arm floaties. It gives you undeserved confidence and will eventually drown you. - DangerBunny Artificial intelligence is the only remedy for natural stupidity. - CDP1802
We have various 'scheduled tasks' that process customer uploaded files or perform sql backups/ftp functions. All of them use custom exes that I wrote. In the case of sql backups, the exe gets started with a job number that links to a text file with configuration info. Why custom executables? 0: the tasks are much too complicated to handle with a batch file 1: security...if I'm decrypting/encrypting/password protecting stuff, I'd hate to have my sensitive info stored in a text file. 2: exes are what I know/deal with every day 3: easy to debug
"Go forth into the source" - Neal Morse
-
No - I use scheduled tasks to run apps, which I write for specific tasks if it's not just command line switches on an existing app. Mostly because I've had too many .BAT files that "just failed" and you can't get much of a clue as to why or even how far it got ... at least with an app you can add progress logging and fault information.
Bad command or file name. Bad, bad command! Sit! Stay! Staaaay... AntiTwitter: @DalekDave is now a follower!
you can still redirect the output of bat files from scheduled tasks, the old
cmd.exe /c start xxx >>log.txt 2>&1
but I agree, properly handling what happens next gets real messy real fast ... way verbose, harder to understand such scripts and no easy way to run in any sort of testing / simulation or debugging mode. (Dare I mention spaghetti code, GOTO included, pretty much becomes SOP to achieve meaningful results.)
This internet thing is amazing! Letting people use it: worst idea ever!
-
Out of curiosity, do any of you (in a professional capacity, as in, the company you work for) have various automated jobs that run at night the still execute batch files? And said jobs are put into a workflow with predecessor / successor dependencies, such that if one of the .bat processes (which can be, say, an EXE or a SQL SP call) fails, the whole dependency chain fails? Or does your company use a more sophisticated way of dealing with automating workflows and dependencies?
Latest Article - Building a Prototype Web-Based Diagramming Tool with SVG and Javascript Learning to code with python is like learning to swim with those little arm floaties. It gives you undeserved confidence and will eventually drown you. - DangerBunny Artificial intelligence is the only remedy for natural stupidity. - CDP1802
The key phrase for me is "at night". I have nightly batch files on each of our department servers that handle backups and other routine stuff. For me, this is an application of KISS. Our automated build process OTOH is a Windows service written in C#. A base class executes all the steps required for the build, from retrieving source code from source control to archiving the build folder at the end in an .ISO file. Product-specific classes derived from the base class handle situations unique to each product. The process itself is fairly granular, and can be paused and resumed or canceled easily. If the service is running on one of our build servers it performs a production build. If it's running on a developer's machine, certain steps are omitted to reduce the build time.
Software Zen:
delete this; -
Out of curiosity, do any of you (in a professional capacity, as in, the company you work for) have various automated jobs that run at night the still execute batch files? And said jobs are put into a workflow with predecessor / successor dependencies, such that if one of the .bat processes (which can be, say, an EXE or a SQL SP call) fails, the whole dependency chain fails? Or does your company use a more sophisticated way of dealing with automating workflows and dependencies?
Latest Article - Building a Prototype Web-Based Diagramming Tool with SVG and Javascript Learning to code with python is like learning to swim with those little arm floaties. It gives you undeserved confidence and will eventually drown you. - DangerBunny Artificial intelligence is the only remedy for natural stupidity. - CDP1802
Batch files are most simple and can be run from scheduled task also and can send out emails, also you can call powershell it a bat file.
Caveat Emptor. "Progress doesn't come from early risers – progress is made by lazy men looking for easier ways to do things." Lazarus Long
-
Out of curiosity, do any of you (in a professional capacity, as in, the company you work for) have various automated jobs that run at night the still execute batch files? And said jobs are put into a workflow with predecessor / successor dependencies, such that if one of the .bat processes (which can be, say, an EXE or a SQL SP call) fails, the whole dependency chain fails? Or does your company use a more sophisticated way of dealing with automating workflows and dependencies?
Latest Article - Building a Prototype Web-Based Diagramming Tool with SVG and Javascript Learning to code with python is like learning to swim with those little arm floaties. It gives you undeserved confidence and will eventually drown you. - DangerBunny Artificial intelligence is the only remedy for natural stupidity. - CDP1802
Marc Clifton wrote:
Or does your company use a more sophisticated way of dealing with automating workflows and dependencies?
cronjobs running bash scripts count?
M.D.V. ;) If something has a solution... Why do we have to worry about?. If it has no solution... For what reason do we have to worry about? Help me to understand what I'm saying, and I'll explain it better to you Rating helpful answers is nice, but saying thanks can be even nicer.
-
Out of curiosity, do any of you (in a professional capacity, as in, the company you work for) have various automated jobs that run at night the still execute batch files? And said jobs are put into a workflow with predecessor / successor dependencies, such that if one of the .bat processes (which can be, say, an EXE or a SQL SP call) fails, the whole dependency chain fails? Or does your company use a more sophisticated way of dealing with automating workflows and dependencies?
Latest Article - Building a Prototype Web-Based Diagramming Tool with SVG and Javascript Learning to code with python is like learning to swim with those little arm floaties. It gives you undeserved confidence and will eventually drown you. - DangerBunny Artificial intelligence is the only remedy for natural stupidity. - CDP1802
No, my current employer prefers to spend more and accomplish less.
-
Out of curiosity, do any of you (in a professional capacity, as in, the company you work for) have various automated jobs that run at night the still execute batch files? And said jobs are put into a workflow with predecessor / successor dependencies, such that if one of the .bat processes (which can be, say, an EXE or a SQL SP call) fails, the whole dependency chain fails? Or does your company use a more sophisticated way of dealing with automating workflows and dependencies?
Latest Article - Building a Prototype Web-Based Diagramming Tool with SVG and Javascript Learning to code with python is like learning to swim with those little arm floaties. It gives you undeserved confidence and will eventually drown you. - DangerBunny Artificial intelligence is the only remedy for natural stupidity. - CDP1802
One of the last places I worked at botched a mainframe migration to J2EE and runs both together. The J2EE works all day queuing data for batch processing. At night the (80's era) scheduler runs the jobs, and if one fails, everything fails. The jobs loads the data needed by the J2EE for the next day. It's a mess, and about half the IT staff is always on-call at night. Oh, and there's no notification system. One person must log in at about 3:00 am to see if everything worked. This is a government agent with a $1.1B budget!
-
Out of curiosity, do any of you (in a professional capacity, as in, the company you work for) have various automated jobs that run at night the still execute batch files? And said jobs are put into a workflow with predecessor / successor dependencies, such that if one of the .bat processes (which can be, say, an EXE or a SQL SP call) fails, the whole dependency chain fails? Or does your company use a more sophisticated way of dealing with automating workflows and dependencies?
Latest Article - Building a Prototype Web-Based Diagramming Tool with SVG and Javascript Learning to code with python is like learning to swim with those little arm floaties. It gives you undeserved confidence and will eventually drown you. - DangerBunny Artificial intelligence is the only remedy for natural stupidity. - CDP1802
Batch files? Absolutely, love 'em, love 'em, love 'em. Can't get my head around PowerShell. This[^] is good, if you want '.bat++'. In fact, it offers a way superior command line to cmd.exe in general.
Paul Sanders http://www.alpinesoft.co.uk
-
Out of curiosity, do any of you (in a professional capacity, as in, the company you work for) have various automated jobs that run at night the still execute batch files? And said jobs are put into a workflow with predecessor / successor dependencies, such that if one of the .bat processes (which can be, say, an EXE or a SQL SP call) fails, the whole dependency chain fails? Or does your company use a more sophisticated way of dealing with automating workflows and dependencies?
Latest Article - Building a Prototype Web-Based Diagramming Tool with SVG and Javascript Learning to code with python is like learning to swim with those little arm floaties. It gives you undeserved confidence and will eventually drown you. - DangerBunny Artificial intelligence is the only remedy for natural stupidity. - CDP1802
Using batch files here. And the batch code is actually pretty sophisticated (at least after I've took it upon me to make it work properly, the Linux guys who created that batch mess really didn't know what they're doing and by their own account, they don't want to know what they're doing).
-
Out of curiosity, do any of you (in a professional capacity, as in, the company you work for) have various automated jobs that run at night the still execute batch files? And said jobs are put into a workflow with predecessor / successor dependencies, such that if one of the .bat processes (which can be, say, an EXE or a SQL SP call) fails, the whole dependency chain fails? Or does your company use a more sophisticated way of dealing with automating workflows and dependencies?
Latest Article - Building a Prototype Web-Based Diagramming Tool with SVG and Javascript Learning to code with python is like learning to swim with those little arm floaties. It gives you undeserved confidence and will eventually drown you. - DangerBunny Artificial intelligence is the only remedy for natural stupidity. - CDP1802
-
Out of curiosity, do any of you (in a professional capacity, as in, the company you work for) have various automated jobs that run at night the still execute batch files? And said jobs are put into a workflow with predecessor / successor dependencies, such that if one of the .bat processes (which can be, say, an EXE or a SQL SP call) fails, the whole dependency chain fails? Or does your company use a more sophisticated way of dealing with automating workflows and dependencies?
Latest Article - Building a Prototype Web-Based Diagramming Tool with SVG and Javascript Learning to code with python is like learning to swim with those little arm floaties. It gives you undeserved confidence and will eventually drown you. - DangerBunny Artificial intelligence is the only remedy for natural stupidity. - CDP1802
We run an overnight process that updates a database with information from a Finance system and this was done using a .NET exe file which is started by Windows Task Scheduler. The exe file accesses the source DB in read only mode and the target DB in update mode. It runs each database update in the right order and provides log files and emailed error, progress and diagnostic reports. I know that if the last report has been emailed then all is well otherwise where it fails is in the logs it leaves behind. It does occasionally fail but that is usually following a target db change that has not been fully tested.
-
Out of curiosity, do any of you (in a professional capacity, as in, the company you work for) have various automated jobs that run at night the still execute batch files? And said jobs are put into a workflow with predecessor / successor dependencies, such that if one of the .bat processes (which can be, say, an EXE or a SQL SP call) fails, the whole dependency chain fails? Or does your company use a more sophisticated way of dealing with automating workflows and dependencies?
Latest Article - Building a Prototype Web-Based Diagramming Tool with SVG and Javascript Learning to code with python is like learning to swim with those little arm floaties. It gives you undeserved confidence and will eventually drown you. - DangerBunny Artificial intelligence is the only remedy for natural stupidity. - CDP1802
You may check WexFlow or may I suggest you Jenkins ?
-
Out of curiosity, do any of you (in a professional capacity, as in, the company you work for) have various automated jobs that run at night the still execute batch files? And said jobs are put into a workflow with predecessor / successor dependencies, such that if one of the .bat processes (which can be, say, an EXE or a SQL SP call) fails, the whole dependency chain fails? Or does your company use a more sophisticated way of dealing with automating workflows and dependencies?
Latest Article - Building a Prototype Web-Based Diagramming Tool with SVG and Javascript Learning to code with python is like learning to swim with those little arm floaties. It gives you undeserved confidence and will eventually drown you. - DangerBunny Artificial intelligence is the only remedy for natural stupidity. - CDP1802
Our company uses [Jenkins](https://jenkins.io/)[[^](https://jenkins.io/)]. Builds can run over night, or be initiated at any time. Jenkins can build out of GitHub, deploy to a VM (for eg.) and report success/failure results to Slack.
Cheers, Mike Fidler "I intend to live forever - so far, so good." Steven Wright "I almost had a psychic girlfriend but she left me before we met." Also Steven Wright "I'm addicted to placebos. I could quit, but it wouldn't matter." Steven Wright yet again.
-
Out of curiosity, do any of you (in a professional capacity, as in, the company you work for) have various automated jobs that run at night the still execute batch files? And said jobs are put into a workflow with predecessor / successor dependencies, such that if one of the .bat processes (which can be, say, an EXE or a SQL SP call) fails, the whole dependency chain fails? Or does your company use a more sophisticated way of dealing with automating workflows and dependencies?
Latest Article - Building a Prototype Web-Based Diagramming Tool with SVG and Javascript Learning to code with python is like learning to swim with those little arm floaties. It gives you undeserved confidence and will eventually drown you. - DangerBunny Artificial intelligence is the only remedy for natural stupidity. - CDP1802
batch files. Kind of. yes sort of. Usually powershell scripts with fairly good transaction processing roll back included that emails successes and failures. We try to avoid the whole Rube Goldberg Machine setup where if one little thing goes wrong the whole contraption doesn't work. It does seem to happen once in awhile. What really chafs me is when it is forced on you by the application vendor. "Oh we have this timer service you setup to run this process. It needs to run this one first and then the second one. It usually takes 15 minutes except when it doesn't. It should be ran at night and no it doesn't email. you just have to manually check it. We recommend setting them to run 2 hours apart. No don't run anything else at that time. Opps sorry that latest patch caused the first one to run for 3 hours. Now everything is messed up. Yes you will need to rerun them during the day. Yes it will be down while this completes. sorry" Yep, we have had that conversation once. Not fun.
To err is human to really mess up you need a computer
-
You may check WexFlow or may I suggest you Jenkins ?
I also agree with using Jenkins and it is free
-
Out of curiosity, do any of you (in a professional capacity, as in, the company you work for) have various automated jobs that run at night the still execute batch files? And said jobs are put into a workflow with predecessor / successor dependencies, such that if one of the .bat processes (which can be, say, an EXE or a SQL SP call) fails, the whole dependency chain fails? Or does your company use a more sophisticated way of dealing with automating workflows and dependencies?
Latest Article - Building a Prototype Web-Based Diagramming Tool with SVG and Javascript Learning to code with python is like learning to swim with those little arm floaties. It gives you undeserved confidence and will eventually drown you. - DangerBunny Artificial intelligence is the only remedy for natural stupidity. - CDP1802
At work we use a product called "Cybermation" , its formal name is "Enterprise Scheduling System (EPS)". At first I hated it, then I thought, hey, it beats 5,000 tasks, or windows services. A long term goal I have, is building something like it. The way it works with Active Directory is nice. It puts a small db on the target prod server, and credentials are cached. In short, something like this is really beneficial, because we have 10's of thousands of jobs running on this. Informatica is also funneled into this, as well as a lot of other big products.
Where there's smoke, there's a Blue Screen of death.
-
Out of curiosity, do any of you (in a professional capacity, as in, the company you work for) have various automated jobs that run at night the still execute batch files? And said jobs are put into a workflow with predecessor / successor dependencies, such that if one of the .bat processes (which can be, say, an EXE or a SQL SP call) fails, the whole dependency chain fails? Or does your company use a more sophisticated way of dealing with automating workflows and dependencies?
Latest Article - Building a Prototype Web-Based Diagramming Tool with SVG and Javascript Learning to code with python is like learning to swim with those little arm floaties. It gives you undeserved confidence and will eventually drown you. - DangerBunny Artificial intelligence is the only remedy for natural stupidity. - CDP1802
At work we use a product called "Cybermation" , its formal name is "Enterprise Scheduling System (EPS)". At first I hated it, then I thought, hey, it beats 5,000 tasks, or windows services. A long term goal I have, is building something like it. The way it works with Active Directory is nice. It puts a small db on the target prod server, and credentials are cached. In short, something like this is really beneficial, because we have 10's of thousands of jobs running on this. Informatica is also funneled into this, as well as a lot of other big products. Another Scheduling product I see i SKYBOT , it looks more modern.
Where there's smoke, there's a Blue Screen of death.
-
Out of curiosity, do any of you (in a professional capacity, as in, the company you work for) have various automated jobs that run at night the still execute batch files? And said jobs are put into a workflow with predecessor / successor dependencies, such that if one of the .bat processes (which can be, say, an EXE or a SQL SP call) fails, the whole dependency chain fails? Or does your company use a more sophisticated way of dealing with automating workflows and dependencies?
Latest Article - Building a Prototype Web-Based Diagramming Tool with SVG and Javascript Learning to code with python is like learning to swim with those little arm floaties. It gives you undeserved confidence and will eventually drown you. - DangerBunny Artificial intelligence is the only remedy for natural stupidity. - CDP1802
I tend to create SQL Server Agent jobs (C# console programs); with a scheduled start. The "dashboard" works well. Jobs can be any type. Easily restarted. No agent for the SQL Server Express version though.
"(I) am amazed to see myself here rather than there ... now rather than then". ― Blaise Pascal
-
Marc Clifton wrote:
And said jobs are put into a workflow with predecessor / successor dependencies, such that if one of the .bat processes (which can be, say, an EXE or a SQL SP call) fails, the whole dependency chain fails?
Automated a build at a previous company, but that excludes the whole chain-fail thingy; you simply check the errorlevel-variable.
Bastard Programmer from Hell :suss: If you can't read my code, try converting it here[^] "If you just follow the bacon Eddy, wherever it leads you, then you won't have to think about politics." -- Some Bell.
We have enough Control-M batch jobs that we have a small team supporting them.
