IBM Mainframe and thousands of millions of ...
-
Hi, I have been asked to write an application to compare about 5,000,000,000 or more records each of which containing at least 2 groups of 14 digit data, and removes records with repetitive numbers. All the data are stored in plain text files. I have also told that I have permission to use an IBM Mainframe computer. I don't know the specifications yet, I just know it has a hard disk with around 7 terabytes of capacity(I have never seen such a thing before:omg:). What I was wondering, was that: - Is there any one who has any experience using these machines? Do I write my application in Win32? or mainframes have different architecture? - How to tune up processing time? Which algorithm do you prefer? (I wish data was stored on a SQL Server and I was going to write just a 'group by') - If I needed to store data, then I prefer using MS. SQL Server. In this case should I buy a special version for the mainframe, or usual (IA32 ) versions will work fine? Thank you very much in advanced.
//This is not a signature while (I'm_alive) { printf("I Love Programming"); }
-
Hi, I have been asked to write an application to compare about 5,000,000,000 or more records each of which containing at least 2 groups of 14 digit data, and removes records with repetitive numbers. All the data are stored in plain text files. I have also told that I have permission to use an IBM Mainframe computer. I don't know the specifications yet, I just know it has a hard disk with around 7 terabytes of capacity(I have never seen such a thing before:omg:). What I was wondering, was that: - Is there any one who has any experience using these machines? Do I write my application in Win32? or mainframes have different architecture? - How to tune up processing time? Which algorithm do you prefer? (I wish data was stored on a SQL Server and I was going to write just a 'group by') - If I needed to store data, then I prefer using MS. SQL Server. In this case should I buy a special version for the mainframe, or usual (IA32 ) versions will work fine? Thank you very much in advanced.
//This is not a signature while (I'm_alive) { printf("I Love Programming"); }
Hamed Mosavi wrote:
Do I write my application in Win32?
Not unless the mainframe is running Windows!
Hamed Mosavi wrote:
or mainframes have different architecture?
If it's an IBM, it could be an AS/400 running OS/400.
"Approved Workmen Are Not Ashamed" - 2 Timothy 2:15
"Judge not by the eye but by the heart." - Native American Proverb
-
Hamed Mosavi wrote:
Do I write my application in Win32?
Not unless the mainframe is running Windows!
Hamed Mosavi wrote:
or mainframes have different architecture?
If it's an IBM, it could be an AS/400 running OS/400.
"Approved Workmen Are Not Ashamed" - 2 Timothy 2:15
"Judge not by the eye but by the heart." - Native American Proverb
There are also many mainframes running different types of unix. depends on the manufacturer and on the administrator.
Don't try it, just do it! ;-)
-
Hamed Mosavi wrote:
Do I write my application in Win32?
Not unless the mainframe is running Windows!
Hamed Mosavi wrote:
or mainframes have different architecture?
If it's an IBM, it could be an AS/400 running OS/400.
"Approved Workmen Are Not Ashamed" - 2 Timothy 2:15
"Judge not by the eye but by the heart." - Native American Proverb
OS/400:wtf: So what compiler can I find there? Is there a C++ compiler available?? I'm a bit afraid. I have never seen AS/400. Does it have a 'sql dbms' equivalent? Do I have to use a version of Oracle? Oh, no. I have not been working with any of these. -- modified at 2:01 Friday 3rd November, 2006
//This is not a signature while (I'm_alive) { printf("I Love Programming"); }
-
OS/400:wtf: So what compiler can I find there? Is there a C++ compiler available?? I'm a bit afraid. I have never seen AS/400. Does it have a 'sql dbms' equivalent? Do I have to use a version of Oracle? Oh, no. I have not been working with any of these. -- modified at 2:01 Friday 3rd November, 2006
//This is not a signature while (I'm_alive) { printf("I Love Programming"); }
Hamed Mosavi wrote:
OS/400 So what compiler can I find there?
My co-worker uses RPG (mostly), Cobol, and C.
Hamed Mosavi wrote:
Is there a C++ compiler available?? Does it have a 'sql dbms' equivalent? Do I have to use a version of Oracle?
I have no idea. All that aside, if you have a good algorithm, what you want can be accomplished on a decent-speed PC.
"Approved Workmen Are Not Ashamed" - 2 Timothy 2:15
"Judge not by the eye but by the heart." - Native American Proverb
-
Hi, I have been asked to write an application to compare about 5,000,000,000 or more records each of which containing at least 2 groups of 14 digit data, and removes records with repetitive numbers. All the data are stored in plain text files. I have also told that I have permission to use an IBM Mainframe computer. I don't know the specifications yet, I just know it has a hard disk with around 7 terabytes of capacity(I have never seen such a thing before:omg:). What I was wondering, was that: - Is there any one who has any experience using these machines? Do I write my application in Win32? or mainframes have different architecture? - How to tune up processing time? Which algorithm do you prefer? (I wish data was stored on a SQL Server and I was going to write just a 'group by') - If I needed to store data, then I prefer using MS. SQL Server. In this case should I buy a special version for the mainframe, or usual (IA32 ) versions will work fine? Thank you very much in advanced.
//This is not a signature while (I'm_alive) { printf("I Love Programming"); }
Hi. Find out what IBM system that you have access to. If it is an S390/ZOS or S390/Z-VM system, then you can easily write an EXEC employing PIPELINES to remove records that contain duplicate data. PIPELINES was specifically designed to handle this kind of situation; processing huge amounts of data easily and efficiently. For more information, check the IBM website for 'CMS PIPELINES', this will tell you everything that you need to know. If you do not have any joy or need further help, let me know. Regards. James. -- modified at 15:19 Friday 3rd November, 2006
-
Hi. Find out what IBM system that you have access to. If it is an S390/ZOS or S390/Z-VM system, then you can easily write an EXEC employing PIPELINES to remove records that contain duplicate data. PIPELINES was specifically designed to handle this kind of situation; processing huge amounts of data easily and efficiently. For more information, check the IBM website for 'CMS PIPELINES', this will tell you everything that you need to know. If you do not have any joy or need further help, let me know. Regards. James. -- modified at 15:19 Friday 3rd November, 2006
Thank you for your informative information. Within next week I'll travel to the city where everything is located. I'll soon check everything out and come back to code project, to ask more questions! Now, I'm tring to find out some information about "CMS PIPELINES". Thank you very much again. (I have heard something about pipe lines already. If I'm not wrong, it was the capability to convert one process to multiple processes and do multiple process simultaneously). Thanks again, Mr. James. -- modified at 22:35 Friday 3rd November, 2006
//This is not a signature while (I'm_alive) { printf("I Love Programming"); }
-
Hamed Mosavi wrote:
OS/400 So what compiler can I find there?
My co-worker uses RPG (mostly), Cobol, and C.
Hamed Mosavi wrote:
Is there a C++ compiler available?? Does it have a 'sql dbms' equivalent? Do I have to use a version of Oracle?
I have no idea. All that aside, if you have a good algorithm, what you want can be accomplished on a decent-speed PC.
"Approved Workmen Are Not Ashamed" - 2 Timothy 2:15
"Judge not by the eye but by the heart." - Native American Proverb
DavidCrow wrote:
All that aside, if you have a good algorithm, what you want can be accomplished on a decent-speed PC.
You might be right, but this is the job that they might do frequently, perhaps once a week. They want to make sure it will not take a long time (More than 1 hour or 2, perhaps). I didn't calculate yet, but I'm sure it's a lot of processes. However I do agree that a good algorithm can convert hours of process to minutes or seconds. I experienced that already! But best algorithms should think a bit when it comes to 5,000,000,000 and more records of data.
//This is not a signature while (I'm_alive) { printf("I Love Programming"); }