Search:
Submit Search


avg. overall data read

Hauptwerk software technical support only. Please make sure you have read the manual, tutorials and FAQ pages before requesting support.

avg. overall data read

Postby gjschipper » Fri Oct 16, 2015 1:38 pm

I have a new PC with a i7-6700 and a SSD Samsung SM951 500GB.
This disk has a Sequential Read of 2150MB/s

With the test tool: as ssd benchmark I can reach the specs, unfortunately I can't reach this speed with Hauptwerk.

The Sample loader: approx. avg. overall data read rate is: 696.49 MB/s.

What can be the bottleneck here?

Logging:
Organ ID: 800459.
Organ ver.: 1.02.
Organ min. Hauptwerk ver.: 4.0.0.000.
Mem.: total raw audio/trem. sample data mem. (excl. pre-proc. audio, images, or other data): 13927.13 MB.
Mem.: overall compression ratio for compressed ranks: 0.00 pct.
Total time: overall: 11.216 sec.
Total time: sample data: 10.348 sec.
Total time: all except sample data: 0.868 sec.
Time: prep.: 0.132 sec.
Time: organ def. loading/parsing: 0.227 sec.
Time: organ def. object/key validation: 0.039 sec.
Time: organ def. prep.: 0.023 sec.
Time: prep. for loading settings: 0.001 sec.
Time: settings loading/parsing: 0.013 sec.
Time: settings object/key validation: 0.003 sec.
Time: settings prep.: 0.001 sec.
Time: data cache validation/prep.: 0.000 sec.
Time: audio data: 10.079 sec.
Time: trem. data: 0.269 sec.
Time: images: 0.119 sec.
Time: comb. and temper. files: 0.310 sec.
Sample loader: loaded from data cache: Y.
Sample loader: data cache total disk size: 7207.30 MB.
Sample loader: buffers: 9.
Sample loader: approx. loader peak mem. usage during audio loading: 169.95 MB.
Sample loader: loader def. mem. usage during audio loading: 135.07 MB.
Sample loader: approx. loader mem. usage during trem. loading: 22.57 MB.
Sample loader: approx. avg. overall data read rate: 696.49 MB/s.
Sample loader: approx. avg. data read rate during disk reader activity: 1832.53 MB/s.
Sample loader: pct. data loading time each stage busy: disk I/O: 38.01.
Sample loader: pct. sample loading time stage busy: 1st proc.: 61.86.
Sample loader: pct. sample loading time stage busy: 2nd proc.: 73.91.
Sample loader: pct. sample loading time stage busy: 3rd proc.: 72.91.
Sample loader: pct. sample loading time stage busy: 4th proc.: 36.39.
Sample loader: pct. sample loading time stage busy: 5th proc.: 95.25.
Sample loader: pct. thread activity due to stage: disk I/O: 10.05.
Sample loader: pct. thread activity due to stage: 1st proc.: 16.35.
Sample loader: pct. thread activity due to stage: 2nd proc.: 19.54.
Sample loader: pct. thread activity due to stage: 3rd proc.: 19.27.
Sample loader: pct. thread activity due to stage: 4th proc.: 9.62.
Sample loader: pct. thread activity due to stage: 5th proc.: 25.18.
gjschipper
Member
 
Posts: 41
Joined: Fri May 11, 2012 3:35 am

Re: avg. overall data read

Postby mdyde » Fri Oct 16, 2015 1:59 pm

The 'avg. data read rate during disk reader activity' is the raw read rate from the SSD, which is close to your optimal SSD benchmark figure anyway:

Sample loader: approx. avg. overall data read rate: 696.49 MB/s.
Sample loader: approx. avg. data read rate during disk reader activity: 1832.53 MB/s.


The 'avg. overall data read rate' figure will be CPU-limited, since Hauptwerk needs to process the data as it reads it from disk (de-compress it, decrypt it, etc.). It uses multiple CPU cores simultaneously, but it will still be CPU-limited, with a very fast SSD. Your figure of around 700 MB/s is actually outstandingly good -- far higher than I've ever seen before!

My 2011 quad-core MacBook Pro manages about 200 MB/s: viewtopic.php?f=16&t=14556#p108156

Hence your PC's loading performance is extremely good, and certainly doesn't indicate a problem.
Best regards,
Martin.

[Please use email or the Contact page if you need to contact us privately, rather than private forum messages.]

Image
User avatar
mdyde
Moderator
 
Posts: 10644
Joined: Fri Mar 14, 2003 2:19 pm
Location: UK

Re: avg. overall data read

Postby gjschipper » Fri Oct 16, 2015 2:26 pm

Hi Martin,

Thanks for your answer.
And I agree with you that the performance is ok, but I think it can be better. Because the CPU is not the bottleneck here. The avg cpu time is around 50%. So there is another bottleneck. Any idea?
gjschipper
Member
 
Posts: 41
Joined: Fri May 11, 2012 3:35 am

Re: avg. overall data read

Postby mdyde » Fri Oct 16, 2015 2:41 pm

The CPU *is* the bottleneck:

Sample loader: pct. data loading time each stage busy: disk I/O: 38.01.
Sample loader: pct. sample loading time stage busy: 1st proc.: 61.86.
Sample loader: pct. sample loading time stage busy: 2nd proc.: 73.91.
Sample loader: pct. sample loading time stage busy: 3rd proc.: 72.91.
Sample loader: pct. sample loading time stage busy: 4th proc.: 36.39.
Sample loader: pct. sample loading time stage busy: 5th proc.: 95.25.


The last of those figures indicates that the relevant thread (CPU core) is running at almost 100%. Hauptwerk's threading model for loading from cache is very highly optimised on the hardware on which we've tested and developed it, but clearly in your case there's scope to increase the threading further in that area (the 5th processing thread').

That's something we'll aim to do in the longer-term (we try to include some performance enhancements in each release, to keep up with evolving hardware), but for now that particular thread (and the performance of the CPU core it's running on) is the bottleneck.
Best regards,
Martin.

[Please use email or the Contact page if you need to contact us privately, rather than private forum messages.]

Image
User avatar
mdyde
Moderator
 
Posts: 10644
Joined: Fri Mar 14, 2003 2:19 pm
Location: UK

Re: avg. overall data read

Postby gjschipper » Fri Oct 16, 2015 3:11 pm

Ok, that's pitty, because Hauptwerk doesn't use all my cores? Or is it using 5 threads?
The overall avg cpu usage is 55% (in taskmanager) so I have resources enough to reach a better performance.

What blocksize is hauptwerk reading the cache file? Maybe the bytes per sector will increase the performance, so I can format my disk in the right blocksize.
Last edited by gjschipper on Thu Feb 04, 2016 3:24 am, edited 1 time in total.
gjschipper
Member
 
Posts: 41
Joined: Fri May 11, 2012 3:35 am

Re: avg. overall data read

Postby gjschipper » Fri Oct 16, 2015 4:55 pm

hmmm, it doesn't matter how to format the disk, because I don't have any latency.
If I disable 4 cores, with the affinity option, I get the best performance:

Code: Select all
Sample loader: approx. avg. overall data read rate: 721.18 MB/s.
Sample loader: approx. avg. data read rate during disk reader activity: 1848.53 MB/s.


I guess Hauptwerk have to use more threads to read the cache into memory. Hopefully we can get it in the next version .... :?:
gjschipper
Member
 
Posts: 41
Joined: Fri May 11, 2012 3:35 am

Re: avg. overall data read

Postby MrNhanduc » Sat Oct 17, 2015 4:10 am

Interesting results. I have had the same question with a Crucial BX100 SSD, reaching 520 MB/s in AS SSD, but only 410 MB/s loading a cache file. The difference seems close (relative) to your SM951 (AHCI or NVME?). My testing was done with a i7-5820K, having 6 cores/12 threads.

Would be great if future updates would get better support for 6 cores when loading?
MrNhanduc
Member
 
Posts: 51
Joined: Thu Feb 12, 2015 7:08 am
Location: The Netherlands

Re: avg. overall data read

Postby mdyde » Sat Oct 17, 2015 4:24 am

gjschipper wrote:I guess Hauptwerk have to use more threads to read the cache into memory. Hopefully we can get it in the next version .... :?:


I can't promise it for any specific version or time-frame, but we do plan to optimise the loading process for more than 5 CPU cores/threads in the future. (It isn't a trivial job.)

For now, and with your existing hardware, you almost certainly won't get significantly higher loading rates than you're seeing. The only way would be to install a processor that had higher per-core performance (e.g. due a faster clock rate).

If any of the 'Sample loader: pct. sample loading time stage busy: ...' metrics are near 100% then the CPU core on which the thread is running is the bottleneck.

Anyway, loading about 14 GB of sample set into memory in just over 10 seconds, as your benchmarks show, is already *extremely* good performance! Enjoy it!
Best regards,
Martin.

[Please use email or the Contact page if you need to contact us privately, rather than private forum messages.]

Image
User avatar
mdyde
Moderator
 
Posts: 10644
Joined: Fri Mar 14, 2003 2:19 pm
Location: UK

Re: avg. overall data read

Postby gjschipper » Wed Apr 27, 2016 6:09 am

Update with other specs (i7-5820K) and an m.2 SSD. I can now reach 1101.87 MB/s.

Code: Select all
Sample loader: approx. avg. overall data read rate: 1101.87 MB/s.
Sample loader: approx. avg. data read rate during disk reader activity: 1619.51 MB/s.
Sample loader: pct. data loading time each stage busy: disk I/O: 68.04.
Sample loader: pct. sample loading time stage busy: 1st proc.: 76.67.
Sample loader: pct. sample loading time stage busy: 2nd proc.: 75.62.
Sample loader: pct. sample loading time stage busy: 3rd proc.: 69.07.
Sample loader: pct. sample loading time stage busy: 4th proc.: 29.29.
Sample loader: pct. sample loading time stage busy: 5th proc.: 97.81.


It's now possible to load Utrecht (Full surround) in 28.3 seconds.

Code: Select all
[Mem. usage stats. MB: Approx. est. of tot. usable phys. mem. remaining: 69899, Approx. est. Hauptwerk sample/obj. mem. (excl. other data): 57983 (0 pageable). Approx. pct. phys. mem. used: 45, OS tot. 'available' phys.: 69899, OS commit tot.: 14, OS tot. page file: 19456, OS tot. virtual: 150428, OS mem. load pct: 46, OS sys. cache: 7, OS process commit: 58388, OS process page faults: 14, OS process work. set: 58287, OS process min. work. set: 74632.]
Organ ID: 000884.
Organ ver.: 2.51.
Organ min. Hauptwerk ver.: 4.0.0.000.
Mem.: total raw audio/trem. sample data mem. (excl. pre-proc. audio, images, or other data): 57183.31 MB.
Mem.: overall compression ratio for compressed ranks: 0.00 pct.
Total time: overall: 28.308 sec.
Total time: sample data: 26.593 sec.
Total time: all except sample data: 1.715 sec.
gjschipper
Member
 
Posts: 41
Joined: Fri May 11, 2012 3:35 am

Re: avg. overall data read

Postby Organorak » Wed May 04, 2016 3:22 pm

mdyde wrote:Anyway, loading about 14 GB of sample set into memory in just over 10 seconds, as your benchmarks show, is already *extremely* good performance! Enjoy it!


What exactly is the problem? I have a six core i7 and a Samsung SSD and fully expect to wait nearly a minute for a 20GB sample set to load!
Organorak
Member
 
Posts: 639
Joined: Sun Nov 22, 2009 7:55 am

Re: avg. overall data read

Postby gjschipper » Thu May 19, 2016 4:22 am

Organorak wrote:What exactly is the problem? I have a six core i7 and a Samsung SSD and fully expect to wait nearly a minute for a 20GB sample set to load!


You have to load uncompressed an use two SSD's in RAID 0.
gjschipper
Member
 
Posts: 41
Joined: Fri May 11, 2012 3:35 am


Return to Technical support

Who is online

Users browsing this forum: No registered users and 5 guests