Very slow loading of sampleset

Hi,

For a few weeks, loading of a sampleset tends to stall. It could take minutes when it took seconds previously.

Computer is running windows 10. The resource monitor when the loading stalls shows that the read rate is in the 30Mb/s range (a bit lower) while the cpu is then mostly idling (core i5 10600K intel). When it is going about normally, it is in the 250Mb/s range (which seems slow, but more acceptable)

The ssd is a 2Tb samsung evo 970 nvme, crystal disk utilities rates it 100% healthy, with read and write speeds in the 3Gb/s range (sequential) which seems quite normal for this drive.

Could someone share some insight as to what the bottleneck could be ?

Btw, I am still using hauptwerk 7, but I see no reason why upgrading to 8 would alleviate the problem.

Pierre-Henri

[mdyde]

Hello Pierre-Henri,

After loading a sample set from cache, if you look at the last INF:2157 message in Hauptwerk's log ("Help | View activity log"), you should be able to see whether the bottleneck is the CPU or drive I/O. E.g. after loading St. Anne's in Hauptwerk v8 on my laptop (which has an i9 CPU and an NVMe SSD) I get:

INF:2157 The organ StAnnesMoseley.Organ_Hauptwerk_xml has been loaded. Metrics:

[Mem. usage stats. MB: Approx. est. of tot. usable phys. mem. remaining: 119482, Approx. est. Hauptwerk sample/obj. mem. (excl. other data): 2338 (0 pageable). Approx. pct. phys. mem. used: 1, OS tot. 'available' phys.: 119482, OS commit tot.: 3, OS tot. page file: 19456, OS tot. virtual: 150128, OS mem. load pct: 8, OS sys. cache: 3, OS process commit: 2628, OS process page faults: 3, OS process work. set: 2640, OS process min. work. set: 18577, OS process max. work. set: 127600.] [Organ data pool mem. enabled: Y, stats.: 1 chunks allocated totalling: 2048 MB (containing 20923 items totalling 1364.469MB), num. non-empty chunks: 1, free space in last non-empty chunk: 683.531 MB, max. free space in any chunk except last non-empty chunk: 0.000 MB, total free space in all chunks except last non-empty chunk: 0.000 MB, chunk sizes MB and quantities: 128 MB: 0, 256 MB: 0, 2048 MB: 1.] [Logical CPU core speeds: current core speeds: 1700-2300 MHz, max core speeds: 2300-2300 MHz.] [CPU core affinity: system: 00FFFFFF, process: 00FFFFFF, queried: Y.]
Organ ID: 000010.
Organ ver.: 6.00.
Organ min. Hauptwerk ver.: 6.0.0.000.
Mem.: total raw audio/trem. sample data mem. (excl. pre-proc. audio, images, or other data): 1294.41 MB.
Mem.: overall compression ratio for compressed ranks: 38.12 pct.
Total time: overall: 1.929 sec.
Total time: sample data: 1.244 sec.
Total time: all except sample data: 0.685 sec.
Time: prep.: 0.249 sec.
Time: organ def. loading/parsing: 0.073 sec.
Time: organ def. object/key validation: 0.046 sec.
Time: organ def. prep.: 0.014 sec.
Time: prep. for loading settings: 0.002 sec.
Time: settings loading/parsing: 0.011 sec.
Time: settings object/key validation: 0.003 sec.
Time: settings prep.: 0.000 sec.
Time: data cache validation/prep.: 0.001 sec.
Time: audio data: 1.235 sec.
Time: trem. data: 0.009 sec.
Time: images: 0.026 sec.
Time: comb. and temper. files: 0.260 sec.
Sample loader: loaded from data cache: Y.
Sample loader: data cache total disk size: 1359.55 MB.
Sample loader: buffers: 16.
Sample loader: approx. loader peak mem. usage during audio loading: 365.76 MB.
Sample loader: loader def. mem. usage during audio loading: 364.82 MB.
Sample loader: approx. loader mem. usage during trem. loading: 52.02 MB.
Sample loader: approx. avg. overall data read rate: 1092.89 MB/s.
Sample loader: approx. avg. data read rate during disk reader activity: 5020.04 MB/s.
Sample loader: pct. data loading time each stage busy: disk I/O: 21.77.
Sample loader: pct. sample loading time stage busy: 1st proc.: 76.45.
Sample loader: pct. sample loading time stage busy: 2nd proc.: 61.22.
Sample loader: pct. sample loading time stage busy: 3rd proc.: 29.84.
Sample loader: pct. sample loading time stage busy: 4th proc.: 90.89.
Sample loader: pct. sample loading time stage busy: 5th proc.: 30.59.
Sample loader: pct. thread activity due to stage: disk I/O: 7.01.
Sample loader: pct. thread activity due to stage: 1st proc.: 24.60.
Sample loader: pct. thread activity due to stage: 2nd proc.: 19.70.
Sample loader: pct. thread activity due to stage: 3rd proc.: 9.60.
Sample loader: pct. thread activity due to stage: 4th proc.: 29.25.
Sample loader: pct. thread activity due to stage: 5th proc.: 9.84.

If the "Sample loader: pct. thread activity due to stage: disk" figure is very high and the "Sample loader: pct. thread activity due to stage ..." figures low then drive I/O is the bottleneck, whereas if the "disk" figure is low-ish then CPU is the bottleneck. If in doubt, post your whole logged INF:2157 message so that we can see all of the figures.

I'd suggest also:

- Make sure that all available motherboard BIOS updates are applied.
- Make sure that all available Windows updates are applied.
- Make sure that all available driver updates are applied.
- Make sure that no drive that Hauptwerk uses is nearly full .
- Make sure that the Windows system drive isn't nearly full.
- Make sure that there's ample free RAM. (E.g. see whether the problem still occurs if using St. Anne's, which is small.)
- As a test, try temporarily disabling your virus scanner, to see whether that's causing the problem.
- Look in Windows Task Manager (Ctr+Alt+Del) on the "Performance" tab to see whether the disks or CPU are showing high activity, even when Hauptwerk isn't loading an organ. If so, look on the "Processes" tab to try to see which process might be responsible.

Btw, I am still using hauptwerk 7, but I see no reason why upgrading to 8 would alleviate the problem.

Hauptwerk v8 should actually give significantly improved loading performance for cases where CPU is the bottleneck (e.g. with modern SSDs that are performing normally). Here's the excerpt from the release notice:

Sample set loading times from cache should be faster than with any previous version. (On recent computers with high-performance CPUs and NVMe SSDs times are typically 30-40% faster than v7.) If upgrading from any previous Hauptwerk version: the first time that you load any given organ in v8, the organ's sample set cache will (re)generate, which may be slow, but subsequent loads will be much faster.
...
Hauptwerk now has the ability to organs unload faster by optionally allocating memory in larger chunks. Two new settings on the "General settings | General preferences | Advanced ..." screen tab control this functionality:

- "Reserve RAM in chunks for fastest organ unloading/loading [may use a little more memory]?", ...
- "Max proportion of computer's RAM to keep reserved when unloading an organ (%)", which allows values 0-65% and defaults to 60%. If this setting is non-zero, when unloading an organ Hauptwerk will keep up to the specified proportion of the computer's RAM allocated to itself, so that any organs that you load subsequently (until you next exit Hauptwerk) can load and unload faster.

However, my guess would be that drive I/O is the problem on your system at the moment, in which case v8 wouldn't help (but it should help once you've solved the current underlying drive performance problem).

Thanks Martin,

Yes, the bottleneck seems to be the ssd.

While testing, with a sampleset downloading somewhat properly, I got
Sample loader: pct. thread activity due to stage: disk I/O: begin around 50, and above 60 when some stalling occurred.

Another line : Sample loader: pct. data loading time each stage busy: disk I/O:
this is around 99.8%

That said, I just got an error message indicating (not sure of the error code) that the cache was corrupted, while loading zutphen sampleset that loaded correctly, albeit quite slowly, yesterday.

I guess a new ssd will be needed...

Weird that the utilities I tried seem to tell me everything is fine with the ssd. Nothing in the SMART status indicates any problem...

[mdyde]

Thanks, Pierre-Henri.

If you haven't done so already, before replacing the SSD I'd still recommend trying the other things I mentioned (check for all updates, try disabling virus scanner, etc.), just in case one of those solves it.

Well, the computer is dedicated to hauptwerk, is offline, and no virus scanner is installed (except some automatic firewall from microsoft). It is quite obvious a lot of updates for the os could be done, but as nothing has changed for this computer to explain the problems I have now, I think it would be a waste of time.

The error I got today raises a red flag, and a new ssd has been ordered...

[mdyde]

Thanks, Pierre-Henri.

Hope the new SSD fully resolves it.

An update to my problem : actually I was wrong writing that nothing had changed in the configuration of my pc, as I added two 32Gb sticks or ram last august. That said, it seemed to work adequately for a few months until the problems appeared.

So I changed the nvme, by cloning it with clonezilla. First try was a failure, as clonezilla froze in the process. I then removed the two newer sticks of ram (the website of the motherboard did not list many approved configurations with four sticks of ram, and mine was not listed, so I had a hunch it could be the reason), and it went faster, and successfully... windows 10 runs without a hitch on the new ssd, as does hauptwerk, and loading of the sampleset is now much faster.

I then tried the ram sticks with memtest86. At least one of the two newer ones is defective. I'll get them replaced, and tested first !

Btw, clonezilla is a great tool for those of us needing to replace a ssd with a bigger one. The process may seem a bit slow (for about 1Tb of data, it took between 1 and 2 hours, having put my source nvme in an 10Gb enclosure), but it runs unsupervised, so it is still a great time saver : installation of windows, drivers, hauptwerk, samplesets, already cached btw in that time...

[mdyde]

Thanks, Pierre-Henri.

Glad to hear it's making progress.