Holding data in memory

[OriginalGriff]

Are we talking about lots and lots of employees? What does the Employee record hold? Anything big, like an image? If not, then I wouldn't worry too much about the space - I'd be more concerned about what happens when the information is changed. If space / time is a real concern, then try a half-way house. Hold info that won't change (and is small) such as name and employee ID / DB primary key in memory, and fetch the rest when you need it. It really is going to depend on what your app is doing: If looking for an employee is always user initiated, then time is sort-of important - a second is generaly acceptable. If it is a batch run, then time is really important and it's worth using memory.

All those who believe in psycho kinesis, raise my hand.

[Bardy85]

The employee record doesn't hold much, about 15 fields, like Name, Surname, Id, etc. There can be up to about 5000 or so Employee records. The app is batch application that runs every 5min. I guess I'm not too worried about holding the data in memory, more about how I'm holding this data. For EG the below class is how I hold and Edit the data. My methods always seem to be public and I don't think that this is the right way of doing it. Enen though it works.

public class EMPLOYEE
{
/// <summary>
/// list of EMPLOYEEs
/// </summary>
public static List<Tools.DB.OL.Xtime.EMPLOYEE> listEMPLOYEE = new List<Tools.DB.OL.Xtime.EMPLOYEE>();

    #region Methods

    /// <summary>
    /// Fills the EMPLOYEE.
    /// </summary>
    public static void FillEMPLOYEE()
    {
        listEMPLOYEE = GenericFactory<Tools.DB.OL.Xtime.EMPLOYEE>.FillList("SELECT \* FROM EMPLOYEE");
    }
    /// <summary>
    /// Inserts the specified new EMPLOYEE.
    /// </summary>
    /// <param name="employee">The employee.</param>
    public static void InsertEMPLOYEE(Tools.DB.OL.Xtime.EMPLOYEE employee)
    {
        GenericFactory<Tools.DB.OL.Xtime.EMPLOYEE>.Insert(ref listEMPLOYEE, employee);
    }
    /// <summary>
    /// Updates the specified new EMPLOYEE.
    /// </summary>
    /// <param name="employee">The employee.</param>
    public static void UpdateEMPLOYEE(Tools.DB.OL.Xtime.EMPLOYEE employee)
    {
        GenericFactory<Tools.DB.OL.Xtime.EMPLOYEE>.Update(ref listEMPLOYEE, employee);
    }
    /// <summary>
    /// Deletes the EMPLOYEE.
    /// </summary>
    /// <param name="employee">The employee.</param>
    public static void DeleteEMPLOYEE(Tools.DB.OL.Xtime.EMPLOYEE employee)
    {
        GenericFactory<Tools.DB.OL.Xtime.EMPLOYEE>.Delete(ref listEMPLOYEE, employee);
    }

    #endregion

    #region SQL Queries

    public static Tools.DB.OL.Xtime.EMPLOYEE FindEMPLOYEE(Predicate<Tools.DB.OL.Xtime.EMPLOYEE> p)
    {
        Tools.DB.OL.Xtime.EMPLOYEE employee = new Tools.DB.OL.Xtime.EMPLOYEE();
        if (listEMPLOYEE.Exists(p))
        {
            employee = listEMPLOYEE.Find(p);
        }
        return employee;

[N a v a n e e t h]

Bardy85 wrote:

Tools.DB.OL.Xtime.EMPLOYEE employee = new Tools.DB.OL.Xtime.EMPLOYEE();
if (listEMPLOYEE.Exists(p))
{
employee = listEMPLOYEE.Find(p);
}

This code is inefficient. listEMPLOYEE.Exists(p) loops through the items in the list to return the result. employee = listEMPLOYEE.Find(p); will again do a iteration on all items and leading into worst performance. You can do something like the following instead.

Tools.DB.OL.Xtime.EMPLOYEE employee = listEMPLOYEE.Find(p); // employee will be null if not found
return employee;

Bardy85 wrote:

There can be up to about 5000 or so Employee records.

How often do you search on this records? If it is quite frequent, your approach is inefficient. Because, list has sequential storage and searches will lead into O(n) complexity. In such cases, querying database will be efficient. If you still need to keep the items in memory, consider a much superior data structure than a normal list. If you have a one to one mapping like, getting employee object from the employee name, use a Dictionary where name will be the key and employee instance will be the value. This will give you constant (O(1)) complexity. Another alternative is to use a sorted list and do binary search for lookup. This will lead into logarithmic complexity and quite efficient with huge recordsets. :)

Best wishes, Navaneeth

[Dan Mos]

Bardy85 wrote:

Tools.DB.OL.Xtime.EMPLOYEE employee = new Tools.DB.OL.Xtime.EMPLOYEE(); if (listEMPLOYEE.Exists(p)) { employee = listEMPLOYEE.Find(p); } return employee;

If you're going to keep it as a in memory List, then I think the best otion/performance is to derive your class from IComparable and then use binary search. something like:

public class Person: IComparable(Person){
private string name="";
private string ocupation="";

public string Name{
  get{return name;}
  set{name=value;}
}
.
.
.
//implement IComparable
public int CompareTo(Person other){
    return this.name.CompareTo(other.name);
}

}

//now somewhere fill a List of Persons
List(Person) persons = Utils.StoredProcs.PersonProcs.GetPersons();

  //and use binary search witch is very fast
  Person p = persons\[
         persons.BinarySearch(new Person{Name="Some Name"})\];
  //Do something with the rest of fields/data
  
  // Binary Search has O(n) = log (n); Of course you need to Order the data
  //I'm retriving it already oreder by the Select(s) statements/procs

I actually implemented a Reporting program this way. It's blazing fast. Faster then our "entity" server. Memory consumption is like 170MB for a hole year worth of data. //And there are tens of milions rows of data from over 10 tables/classes. Botom line: if your app is to be run by a well established nr of users within your "entity"/firm/enterprise/whatever then go for it. Try using StoredProcs instead of string commands => significant speed gains.

[Bardy85]

I currently do implement IComparable, but don't sort the list of Objects or Use BinarySearch. I guess this would speed up things up. Only problem I see is that BinarySearch does not take a Predicate as an arguement, and I use Predicates to find a matching object. Do you know another way arround this? For example my Employee object has quite a few fields and I don't always want to search where the whole employee object matches, only maybe ID or Lastname, thats why I like using Predicates. In your above example you can only search on Name. I guess you can't do a binary search and then use Predicate to find that specific employee object, because of the way a binary sort works. Anyway thanks for you comments. They have shed some light on my problems. Thanks.

[Bardy85]

Mos Dan - Lucian wrote:

Try using StoredProcs instead of string commands => significant speed gains.

I've read that StoredProcs arn't quicker then normal queries. Have I read wrong? http://betav.com/blog/billva/2006/05/are_stored_procedures_faster_t.html[^]

[Luc Pattyn]

FWIW: I don't use predicates, but this little article[^] explains a lot about user-defined sorting of collections (you can't do smart searches such as BinarySearch unless your collection is sorted on the relevant criterium). :)

Luc Pattyn [Forum Guidelines] [Why QA sucks] [My Articles]

---

I only read code that is properly formatted, adding PRE tags is the easiest way to obtain that.

---

[Dan Mos]

Now I read it too:). But there are many places where they say SP are faster and I tend to belive them. I'm no SQL guru but this is how I think of the process: 1) NO SP: => the SQL engine recieves a string/guery that needs to be compiled. So it compiles the string query and then it performs the actual query 2) SP: => it directly executes the query

[Bardy85]

Agreed. I think then i'll have to sort the List on the field thats going to be most used. Then I can sort the list and use a binary search on that criterium. If I have to search on any other field I'll just have to use the standard search, which I think is a bubble search. Anyway thanks.

[Dan Mos]

Bardy85 wrote:

I currently do implement IComparable, but don't sort the list of Objects or Use BinarySearch. I guess this would speed up things up. Only problem I see is that BinarySearch does not take a Predicate as an arguement, and I use Predicates to find a matching object. Do you know another way arround this? For example my Employee object has quite a few fields and I don't always want to search where the whole employee object matches, only maybe ID or Lastname, thats why I like using Predicates. In your above example you can only search on Name.

True. But it totally depends on what you want to do with the data. In my case there is just one table/class that needs to be sorted on different filed(s) for different reports. One way to use Binary search no matter the field or fields is to sort the List proving a custom IComparer each time. This makes sence ONLY IF after you sort the list you do a lot of searching using that comparer cause the Sort time is O(n) = n* log(n) while a LINQ/Predicate is O(n)=n, but if you do a lot of searching after sorting by the "ocupation" filed for example, then the extra time needed for sorting is totally worth the while. Hope you get what I tried to say.

modified on Saturday, January 16, 2010 5:40 PM