We have a server that is pulling a bazillion rows via JDBC (I think) daily.>> What is the fastest way to pull this data?
No magic wand for a classic O(n) complexity that keep increasing for a "truncate and rebuild the world" file transfer.It's a I/O bound problem.
If you want to keep the process as is you need to look and network and source and destination I/O use, basically all the I/O chain and increase any bottleneck.But if it now runs in 3h instead of 4h , it is significant to the problem or solve it? Maybe 4h is fine. The problem is rate of change. Will tomorrow be 6h because it is cumulative and doesn't fit the night window and not my problem anymore when it will be ;D ?
Software wise: enable batch writing on INSERT and I would disable any form of commit/transaction.If it is just record transfer, on the target disable all indexes, then rebuild them after transfer.
Put an index on the source if you really need to fetch with ordering.
bye
This mailing list archive is Copyright 1997-2025 by midrange.com and David Gibbs as a compilation work. Use of the archive is restricted to research of a business or technical nature. Any other uses are prohibited. Full details are available on our policy page. If you have questions about this, please contact
[javascript protected email address].
Operating expenses for this site are earned using the Amazon Associate program and Google Adsense.
midrange.com
