Friday Programming Quiz [modified]

[Daniel Grunwald]

My first solution requires loading everything into memory (ReadAllLines/WriteAllLines uses arrays, OrderBy requires having the whole list in memory). Here is another LINQ solution that uses a custom functions for reading/writing files and merging the enumerables:

using System;
using System.IO;
using System.Collections.Generic;
using System.Linq;
static class Program {
public static void Main(string[] args) {
SortedMerge(args.Select(
fileName => FileReadLines(fileName).Select(
line => new { ID = int.Parse(line.Split(',')[0]), Name = line.Split(',')[1]}
)), a=>a.ID)
.Select(a=>string.Format("{0},{1}", a.ID, a.Name))
.WriteToFile("output.txt");
}

static IEnumerable<string> FileReadLines(string fileName)
{
	using (StreamReader reader = new StreamReader(fileName)) {
		string line;
		while ((line = reader.ReadLine()) != null) {
			Console.WriteLine("read " + line + " from " + fileName);
			yield return line;
		}
	}
}

static void WriteToFile(this IEnumerable<string> lines, string fileName)
{
	using (StreamWriter writer = new StreamWriter(fileName)) {
		foreach (string line in lines) {
			Console.WriteLine("write " + line + " to " + fileName);
			writer.WriteLine(line);
		}
	}
}

static IEnumerable<T> SortedMerge<T, K>(IEnumerable<IEnumerable<T>> inputs, Func<T, K> keySelector) where K : IComparable<K>
{
	var enumerators = inputs.Select(o=>o.GetEnumerator()).ToList();
	var disposables = enumerators.ToList(); // make copy of enumerators for disposing them later
	try {
		// move all enumerators on the first element
		enumerators.RemoveAll(e=>!e.MoveNext());
		while (enumerators.Count > 0) {
			int smallest = 0;
			for (int i = 1; i < enumerators.Count; i++) {
				// the the element of the current enumerator smaller than the best found so far?
				if (keySelector(enumerators\[i\].Current).CompareTo(keySelector(enumerators\[smallest\].Current)) < 0) {
					smallest = i;
				}
			}
			yield return enumerators\[smallest\].Current;
			if (!enumerators\[smallest\].MoveNext())
				enumerators.RemoveAt(smallest);
		}
	} finally {
		disposables.ForEach(d => d.Dispose());
	}
}

}

The debug output shows that the program is writing as soon as possible; written lines will be collected by the GC, so this solution can merge multi-GB files without running out of memory. Output:

C:\temp\SharpDevelop Projects\CPQ

[Tarakeshwar Reddy]

Dictionary<int, string> cpians = new Dictionary<int, string>();
List<int> keylist = new List<int>();

string strLine = "";

foreach (System.IO.FileInfo fi in new System.IO.DirectoryInfo(@"C:\quiz\").GetFiles())
{              
    using (System.IO.StreamReader sr = fi.OpenText())
    {
        while (sr.Peek() >= 0)
        {
            strLine = sr.ReadLine();
            cpians.Add(Int32.Parse(strLine.Split(',')[0]), strLine.Split(',')[1]);
            keylist.Add(Int32.Parse(strLine.Split(',')[0]));
        }
    }
}

keylist.Sort();

using (System.IO.FileStream file = new System.IO.FileStream(@"C:\TotalList.txt", System.IO.FileMode.Create, System.IO.FileAccess.Write))
{
    using (System.IO.StreamWriter sw = new System.IO.StreamWriter(file))
    {
        foreach (int key in keylist)
        {
            sw.WriteLine(key + "," + cpians[key]);
        }
    }
}

The items in the file need not be sorted for this solution.

[PIEBALDconsult]

string inline ;

System.Collections.Generic.List list =
new System.Collections.Generic.List() ;

while ( ( inline = System.Console.ReadLine() ) != null )
{
list.Add ( inline.PadLeft ( inline.Length + 16 - inline.IndexOf ( ',' ) , ' ' ) ) ;
}

list.Sort() ;

foreach ( string s in list )
{
System.Console.WriteLine ( s.TrimStart() ) ;
}

[lost in transition]

I would like to see someone right a solution in QBasic:) or Pascal:rose:. Make you own front end. Will this work.

namespace CP
{
static class Program
{
/// /// The main entry point for the application.
///
[STAThread]
static void Main()
{
PQuizOTD pqtd = new PQuizOTD();
pqtd.ReadFile(@"C:\TestData1.txt");
pqtd.ReadFile(@"C:\TestData2.txt");
pqtd.ExportToFile();
}
}
class PQuizOTD
{
private DataTable dt;

    public PQuizOTD()
    {
        dt = new DataTable();
        dt.Columns.AddRange(
                    new DataColumn\[\] { 
                        new DataColumn("ID"), 
                        new DataColumn("Name") });
    }

    public void ReadFile(string FileName)
    {
        string line;
        StreamReader sr = new StreamReader(FileName);
        while (!sr.EndOfStream)
        {
            line = sr.ReadLine();
            int index = line.IndexOf(",");
            DataRow dr = dt.NewRow();
            dr\[0\] = line.Substring(0, index);
            dr\[1\] = line.Substring(index + 1, (line.Length - index - 1));
            dt.Rows.Add(dr);
        }
    }

    public void ExportToFile()
    {
        dt.DefaultView.Sort = "Name ASC";
        dt.AcceptChanges();
        dt = dt.DefaultView.ToTable();
        StreamWriter sw = new StreamWriter(@"C:\\TestData\_Export.txt");
        for (int i = 0; i < dt.Rows.Count; i++)
        {
            sw.WriteLine(dt.Rows\[i\]\[0\].ToString() + "," + dt.Rows\[i\]\[1\].ToString());
        }
        sw.Close();
    }
}

}

---

God Bless, Jason
God doesn't believe in atheist but He still loves them.

[David Crow]

Looks like a job for merge sort.

---

"A good athlete is the result of a good and worthy opponent." - David Crow

"To have a respect for ourselves guides our morals; to have deference for others governs our manners." - Laurence Sterne

[David Crow]

Are you sure that TYPE works with wildcards?

---

"A good athlete is the result of a good and worthy opponent." - David Crow

"To have a respect for ourselves guides our morals; to have deference for others governs our manners." - Laurence Sterne

[Tarakeshwar Reddy]

Modified version using SortedList<>.

SortedList<int, string> cpians = new SortedList<int, string>();                       
string strLine = "";

foreach (System.IO.FileInfo fi in new System.IO.DirectoryInfo(@"C:\quiz\").GetFiles())
{
    using (System.IO.StreamReader sr = fi.OpenText())
    {
        while (sr.Peek() >= 0)
        {
            strLine = sr.ReadLine();
            cpians.Add(Int32.Parse(strLine.Split(',')[0]), strLine.Split(',')[1]);                       
        }
    }
}

using (System.IO.FileStream file = new System.IO.FileStream(@"C:\TotalList.txt", System.IO.FileMode.Create, System.IO.FileAccess.Write))
{
    using (System.IO.StreamWriter sw = new System.IO.StreamWriter(file))
    {
        foreach (KeyValuePair<int, string> kvp in cpians)
        {
            sw.WriteLine(kvp.Key + "," + kvp.Value);
        }
    }
}

[Daniel Grunwald]

Take a look at my "on-the-fly" solution for a merge sort solution that requires memory only linear to the number of files, independent from the file size.

[martin_hughes]

I think I need to adopt this style of programming :)

"It was the day before today.... I remember it like it was yesterday." -Moleman

[martin_hughes]

Surely you'd only get a suffusion of yellow given the inputs?

"It was the day before today.... I remember it like it was yesterday." -Moleman

[Rama Krishna Vavilala]

Infact you don't even need fi.OpenText. You can directly supply the file names to the StreamReader and StreamWriter.

Co-Author ASP.NET AJAX in Action

[PIEBALDconsult]

Yes, it works.

[Tarakeshwar Reddy]

Oh yeah, for the StreamWriter, I could have directly given the filename :doh:. In the StreamReader, I use FileInfo to get a list of files in the particular directory, since I have it declared I am using fi.OpenText() which returns to me a StreamReader

[Rama Krishna Vavilala]

I guess you are the winner. Will you like a copy of ASP.NET AJAX in Action?

Co-Author ASP.NET AJAX in Action

[Anthony Potts]

that's not funny, I mainly do COBOL. It makes me sad to think about how this would be done in COBOL.

"If you really want something in this life, you have to work for it. Now, quiet! They're about to announce the lottery numbers..." - Homer Simpson

[PIEBALDconsult]

Picard: "Data, do something with these files."