avg. overall data read

I have a new PC with a i7-6700 and a SSD Samsung SM951 500GB.
This disk has a Sequential Read of 2150MB/s

With the test tool: as ssd benchmark I can reach the specs, unfortunately I can't reach this speed with Hauptwerk.

The Sample loader: approx. avg. overall data read rate is: 696.49 MB/s.

What can be the bottleneck here?

Logging:
Organ ID: 800459.
Organ ver.: 1.02.
Organ min. Hauptwerk ver.: 4.0.0.000.
Mem.: total raw audio/trem. sample data mem. (excl. pre-proc. audio, images, or other data): 13927.13 MB.
Mem.: overall compression ratio for compressed ranks: 0.00 pct.
Total time: overall: 11.216 sec.
Total time: sample data: 10.348 sec.
Total time: all except sample data: 0.868 sec.
Time: prep.: 0.132 sec.
Time: organ def. loading/parsing: 0.227 sec.
Time: organ def. object/key validation: 0.039 sec.
Time: organ def. prep.: 0.023 sec.
Time: prep. for loading settings: 0.001 sec.
Time: settings loading/parsing: 0.013 sec.
Time: settings object/key validation: 0.003 sec.
Time: settings prep.: 0.001 sec.
Time: data cache validation/prep.: 0.000 sec.
Time: audio data: 10.079 sec.
Time: trem. data: 0.269 sec.
Time: images: 0.119 sec.
Time: comb. and temper. files: 0.310 sec.
Sample loader: loaded from data cache: Y.
Sample loader: data cache total disk size: 7207.30 MB.
Sample loader: buffers: 9.
Sample loader: approx. loader peak mem. usage during audio loading: 169.95 MB.
Sample loader: loader def. mem. usage during audio loading: 135.07 MB.
Sample loader: approx. loader mem. usage during trem. loading: 22.57 MB.
Sample loader: approx. avg. overall data read rate: 696.49 MB/s.
Sample loader: approx. avg. data read rate during disk reader activity: 1832.53 MB/s.
Sample loader: pct. data loading time each stage busy: disk I/O: 38.01.
Sample loader: pct. sample loading time stage busy: 1st proc.: 61.86.
Sample loader: pct. sample loading time stage busy: 2nd proc.: 73.91.
Sample loader: pct. sample loading time stage busy: 3rd proc.: 72.91.
Sample loader: pct. sample loading time stage busy: 4th proc.: 36.39.
Sample loader: pct. sample loading time stage busy: 5th proc.: 95.25.
Sample loader: pct. thread activity due to stage: disk I/O: 10.05.
Sample loader: pct. thread activity due to stage: 1st proc.: 16.35.
Sample loader: pct. thread activity due to stage: 2nd proc.: 19.54.
Sample loader: pct. thread activity due to stage: 3rd proc.: 19.27.
Sample loader: pct. thread activity due to stage: 4th proc.: 9.62.
Sample loader: pct. thread activity due to stage: 5th proc.: 25.18.

[mdyde]

The 'avg. data read rate during disk reader activity' is the raw read rate from the SSD, which is close to your optimal SSD benchmark figure anyway:

Sample loader: approx. avg. overall data read rate: 696.49 MB/s.
Sample loader: approx. avg. data read rate during disk reader activity: 1832.53 MB/s.

The 'avg. overall data read rate' figure will be CPU-limited, since Hauptwerk needs to process the data as it reads it from disk (de-compress it, decrypt it, etc.). It uses multiple CPU cores simultaneously, but it will still be CPU-limited, with a very fast SSD. Your figure of around 700 MB/s is actually outstandingly good -- far higher than I've ever seen before!

My 2011 quad-core MacBook Pro manages about 200 MB/s: viewtopic.php?f=16&t=14556#p108156

Hence your PC's loading performance is extremely good, and certainly doesn't indicate a problem.

Hi Martin,

Thanks for your answer.
And I agree with you that the performance is ok, but I think it can be better. Because the CPU is not the bottleneck here. The avg cpu time is around 50%. So there is another bottleneck. Any idea?

[mdyde]

The CPU *is* the bottleneck:

Sample loader: pct. data loading time each stage busy: disk I/O: 38.01.
Sample loader: pct. sample loading time stage busy: 1st proc.: 61.86.
Sample loader: pct. sample loading time stage busy: 2nd proc.: 73.91.
Sample loader: pct. sample loading time stage busy: 3rd proc.: 72.91.
Sample loader: pct. sample loading time stage busy: 4th proc.: 36.39.
Sample loader: pct. sample loading time stage busy: 5th proc.: 95.25.

The last of those figures indicates that the relevant thread (CPU core) is running at almost 100%. Hauptwerk's threading model for loading from cache is very highly optimised on the hardware on which we've tested and developed it, but clearly in your case there's scope to increase the threading further in that area (the 5th processing thread').

That's something we'll aim to do in the longer-term (we try to include some performance enhancements in each release, to keep up with evolving hardware), but for now that particular thread (and the performance of the CPU core it's running on) is the bottleneck.

Ok, that's pitty, because Hauptwerk doesn't use all my cores? Or is it using 5 threads?
The overall avg cpu usage is 55% (in taskmanager) so I have resources enough to reach a better performance.

What blocksize is hauptwerk reading the cache file? Maybe the bytes per sector will increase the performance, so I can format my disk in the right blocksize.

hmmm, it doesn't matter how to format the disk, because I don't have any latency.
If I disable 4 cores, with the affinity option, I get the best performance:

Code: Select all

Sample loader: approx. avg. overall data read rate: 721.18 MB/s.
Sample loader: approx. avg. data read rate during disk reader activity: 1848.53 MB/s.

I guess Hauptwerk have to use more threads to read the cache into memory. Hopefully we can get it in the next version ....

Interesting results. I have had the same question with a Crucial BX100 SSD, reaching 520 MB/s in AS SSD, but only 410 MB/s loading a cache file. The difference seems close (relative) to your SM951 (AHCI or NVME?). My testing was done with a i7-5820K, having 6 cores/12 threads.

Would be great if future updates would get better support for 6 cores when loading?

[mdyde]

I guess Hauptwerk have to use more threads to read the cache into memory. Hopefully we can get it in the next version ....

I can't promise it for any specific version or time-frame, but we do plan to optimise the loading process for more than 5 CPU cores/threads in the future. (It isn't a trivial job.)

For now, and with your existing hardware, you almost certainly won't get significantly higher loading rates than you're seeing. The only way would be to install a processor that had higher per-core performance (e.g. due a faster clock rate).

If any of the 'Sample loader: pct. sample loading time stage busy: ...' metrics are near 100% then the CPU core on which the thread is running is the bottleneck.

Anyway, loading about 14 GB of sample set into memory in just over 10 seconds, as your benchmarks show, is already *extremely* good performance! Enjoy it!

Update with other specs (i7-5820K) and an m.2 SSD. I can now reach 1101.87 MB/s.

Code: Select all

Sample loader: approx. avg. overall data read rate: 1101.87 MB/s.
Sample loader: approx. avg. data read rate during disk reader activity: 1619.51 MB/s.
Sample loader: pct. data loading time each stage busy: disk I/O: 68.04.
Sample loader: pct. sample loading time stage busy: 1st proc.: 76.67.
Sample loader: pct. sample loading time stage busy: 2nd proc.: 75.62.
Sample loader: pct. sample loading time stage busy: 3rd proc.: 69.07.
Sample loader: pct. sample loading time stage busy: 4th proc.: 29.29.
Sample loader: pct. sample loading time stage busy: 5th proc.: 97.81.

It's now possible to load Utrecht (Full surround) in 28.3 seconds.

Code: Select all

[Mem. usage stats. MB: Approx. est. of tot. usable phys. mem. remaining: 69899, Approx. est. Hauptwerk sample/obj. mem. (excl. other data): 57983 (0 pageable). Approx. pct. phys. mem. used: 45, OS tot. 'available' phys.: 69899, OS commit tot.: 14, OS tot. page file: 19456, OS tot. virtual: 150428, OS mem. load pct: 46, OS sys. cache: 7, OS process commit: 58388, OS process page faults: 14, OS process work. set: 58287, OS process min. work. set: 74632.]
Organ ID: 000884.
Organ ver.: 2.51.
Organ min. Hauptwerk ver.: 4.0.0.000.
Mem.: total raw audio/trem. sample data mem. (excl. pre-proc. audio, images, or other data): 57183.31 MB.
Mem.: overall compression ratio for compressed ranks: 0.00 pct.
Total time: overall: 28.308 sec.
Total time: sample data: 26.593 sec.
Total time: all except sample data: 1.715 sec.

Anyway, loading about 14 GB of sample set into memory in just over 10 seconds, as your benchmarks show, is already *extremely* good performance! Enjoy it!

What exactly is the problem? I have a six core i7 and a Samsung SSD and fully expect to wait nearly a minute for a 20GB sample set to load!

What exactly is the problem? I have a six core i7 and a Samsung SSD and fully expect to wait nearly a minute for a 20GB sample set to load!

You have to load uncompressed an use two SSD's in RAID 0.