Question : Fastest Way to Store/Retrieve Data

I have a question about data storage/retrieval methods.

I am developing a multi-threaded C# application that will be dealing with very large arrays of data objects.  Due to the size and number of these data objects, it would be highly impractical to store them in process memory and the objects will also have to be available across processes/threads.  I want to put a representation (serialization, etc) of the objects into a data store that can be paged to fetch the data object representation as needed.  The processes/threads will be running on the same machine as the data store, so you can assume that there won't be any delays due to different environments.  I'm curious what would be the fastest data storage method for storing and retrieving the items?  I wasn't sure if using an XML file, or just plain text would be better than using a database to store the objects, or if there was something else that would be better than any of those.

Answer : Fastest Way to Store/Retrieve Data

Take a look at db4o (an object database) it is pretty easy to use and also very fast.

http://www.db4o.com/

If you are gpl'ing your software it is also free. You would want to most likely use the in-process one but it can also easily scale to support multiple machines / processes for you transparently.

Beyond that I am curious how often you are writing data as opposed to reading data ... is it write once then read N times? If so it might be even faster for you to just use binary serialization to a file with an index.

Cheers,

Greg