Re: Image Catalog over NFS

Hi Patrick,


On 10/12/24 08:57, Patrik Schindler wrote:

Hello Diego,

Am 09.12.2024 um 22:35 schrieb Diego E. KESSELMAN<diegokesselman@xxxxxxxxx>:

I have found that GO SAVE->22 is not able to run the FTP command, because of the small commands set, so I added this 2 libraries to make it functional.

Curiosity question: Why did you need to run the FTP command from a limited restore environment?

Sorry, I was not clear. I usually save this to an Optical Remote Image Catalog , and the remaining data , using Option 23, to a Virtual Tape Image Catalog (using local disk), when migrating IBM i to the cloud, or backing up data from On-Prem to the cloud.

Then...

* Compress the Virtual Tape Images using ZSTD (faster than PIGZ)
* Upload to the Object Storage (usually IBM Cloud Object Storage, WASABI or Azure BLOB)
* Download from the Object Storage to my Linux local to my DR or Target IBM i
* Uncompress the Tape Images
* Install the Target IBM i OS from TFTP
* Restore both libraries
* Download with FTP from Linux to Target IBM i
* Load the images in my Image Catalog
* Restore the data using GO RESTORE -> 23

But the option 21 works, slow but works.
Slow in which regard? IPL? Restore?

Saving NFS V3 on UDP is slow.

Uhm. I can't duplicate this claim, from decades long own experience.

However, I know that NFS over UDP is much more sensitive to network latency, compared to TCP. It was once common knowledge to never run NFS across routers, because the routers in the late 1980's introduced quite a few ms of latency.

I have found that NFS V3 over UDPs needs a lot of RAM and CPU to work properly. I have compared copying a file with NFS vs SCP and FTP. And when using iSCSI (VTL) for saving, the difference is evident.

If you compare saving on NFS vs iSCSI VTL (Falconstor) , the VTL is really faster.

This observation can base on many reasons and side effects, not just UDP being (perceivably) inferior. In addition, comparing iSCSI — being a block oriented protocol — with NFS — being a file oriented protocol — seems to me a bit like comparing apples to oranges.

If you compare NFS V3 with UDP vs NFS V4 or V4.1 with TCP, there is a big change. Unfortunately you cannot use NFS V4.x over TCP for remote install your IBM i.

I know it's like comparing rocks and trees, but both are used for the same purpose: Network Backup/Restore

But... If you have a fast 10Gb network with jumbo frames, and a target Linux with "more than 32GB of RAM", SSD and fast CPU, you can get a decent transfer rate.

Jumbo frames are a no go in my scenario, because NFS based image catalogs only utilize UDP through the service adapter code. I'm not aware about the possibility to configure the MTU size for the service tools adapter in 7.3, so it's fixed at 1500 Bytes. UDP has no payload negotiation abilities, compared to TCP which negotiates the maximum segment size during connection setup. Thus, answers from the NFS server which are larger than 1500 Bytes are dropped by the service tools adapter as invalid, oversized frames.

On the other hand, I was astonished about the transfer rate obtainable to an older machine as NFS server over "just" 1GbE. Not too precise measuring from first write to the empty image to the image to completion indication in the message line calculate to roughly 33 MBytes/s. This includes "wait time" between the individual calls to the data/file system specific sav commands. I'm pretty sure that the limitation is largely the target Linux machine's I/O subsystem.

Old Debian and Ubuntu Server releases used NFS with UDP

How old is "old"? And what do you precisely mean with "used"? Are we talking about the kernel-based NFS server or the client — being kernel based, because mounting filesystems is handled in the kernel —, or both?

The Kernel-NFS-Server is one option, but there are other NFS servers available. Sorry, I can't remember when was the NFS switch from UDP to TCP, probably on Debian 11. With CentOS/RedHat the change came with V8.x

And yes, you can change values and parameters to make it work.

I will certainly procrastinate automating the Save 21, though. This is handled by QMNSAVE CLP, which gets parameters in one large variable from the menu (I presume), and also calls further CL programs. Too much of a tangle for me currently.

We don't use backup to NFS regularly, only on migrations.

When running in the cloud the Option 21 lose its value, because you have snapshots, clones and exports. We usually save to local disk and upload to S3 buckets once we have compressed everything.

Our next challenge is to compress with a second job monitoring media changes, so we can reduce the amount of data on disk, improve compression ratio and improve backup job times.