Re: How to handle Big Data under IBM i?

From: Matt Olson
Use the right API and you'll be golden.

Okay, but are you declining to explain the difference between standard ODBC / JDBC / .Net Provider interfaces and the one you used?

Simple answer here is your using the wrong tool for the job.

I believe that remains to be seen. I for one don't fully understand the API you used nor how it might apply to Big Data. Say you have devices that are recording measurements and feeding unstructured data to a computer and you want to store it in a database for future analysis. Would the API you referenced be the right tool for the job?

I sure bet you were surprised when I could show how you can
get the same type of performance in a more versatile way
(cross database capable) and that this functionality is not
something inherently unique to IBM i.

Surprised? Well, puzzled would be more accurate. I began this journey by reviewing the documentation and studying the architecture of memSQL and VoltDB, so I'm not "surprised" by high performance. But in this case, I don't fully understand the interface used in your benchmark.

Consider the following clip of your code:

1.       for (int i = 1000000; i <= 130000000; i += 1000000)
2.       {
3.         loadTest.Flush();
4.       }
5.

Does loadTest.Flush() queue work that runs asynchronously in a separate thread, then returns (like submitting a batch job)?

What would happen if you ran an SQL Select count(*) from LoadTest immediately following your "for" loop? Would it return 130,000,000?

-Nathan.