HW 8 - CPU spikes, drop-outs and crackling - new approach?

(Martin, if you feel this is related enough to a past forum thread on the spikes, drop-outs, crackling issue, feel free to reference or recategorize)

Recently I updated to Hauptwerk 8 on a dual processor with more than 16 cores running Windows 10. Since, I have noticed many CPU spikes, audio drop-outs and audio crackling on nearly all sample sets, including St. Annes. I have been following Mark N's debug adventure (viewtopic.php?f=3&t=21086) with a lot of interest.

Curiously, many sample sets running HW 8 on another Hauptwerk system with a nearly 10 year old Intel quad-core processor running Windows 10 rarely ever has any CPU spikes, drop-outs or crackling. I have checked both systems and their Hauptwerk and Windows 10 OS settings appear nearly identical.

I have used Cantabile's Glitch Free - An in-depth guide to tuning Windows for reliable real-time audio performance - https://download.cantabilesoftware.com/GlitchFree.pdf. Despite configuring numerous tuning suggestions (including running as Administrator at Real-time priority and for Hauptwerk, Set Affinity for all but the first 4 cores), the dual proc system still spikes, drops out, and crackles at least once a minute, sometimes multiple times per minute.

The main approach has been to eliminate all other CPU spiking apps, drivers and firmware, etc. Last night, it dawned on me that there might a useful opposite approach:
1) try to get Hauptwerk to run only on some "out of the way" cores
or to the logical extreme:
2) try to dedicate some cores to exclusively run Hauptwerk only

For 1) run only on some "out of the way" cores, I went to Task Manager / Processes tab:
a. selected Hauptwerk, Right-click Go to details
b. Set Affinity, deselected all, and then selected only 6-8 of the highest odd numbered (some even prime numbered, for grins) cores.

The intent of selecting "out of the way" cores was to select cores least likely to be selected by any non-Hauptwerk processes/threads, including from Windows OS (avoid Core 0 or 1, avoid Core N or Core N-1, avoid even-numbered cores). Yes, this is at best an educated guess, at worst a WAG.

Surprisingly, the frequency of CPU spikes, audio drop-outs and audio crackling dropped to only 1 in the next hour of playing several small (<5 GB), medium (<16 GB) and large (~40-50GB) sample sets. Admittedly, a few hours later, the frequency increased to 1 every several minutes. This is progress, but clearly not infrequent enough.

Towards 2), I found a Microsoft article on CPU Sets on Windows - https://learn.microsoft.com/en-us/windo ... d/cpu-sets.

CPU Sets provide APIs to declare application affinity in a 'soft' manner that is compatible with OS power management. Additionally, the API provides applications with the ability to reaffinitize all background threads in the process to a subset of processors using the Process Default mechanism to avoid interference from OS threads within the process.

It does not appear clear whether CPU Sets could actually dedicate some cores exclusively to an application such as Hauptwerk, or are CPU Sets only another way of programmatically setting Affinities.

A further excerpt:

Some versions of Windows support Core Reservation policies, in which a subset of the system’s CPU Sets can be devoted to the exclusive use of individual applications and workloads.

I worry that Core Reservation is only supported on Windows Server OS and not on Windows Professional (which many Hauptwerk PCs use).

I just thought I would mention the opposite approach idea, in case Martin hasn't considered this for a Hauptwerk implementation enhancement or other Hauptwerk PC users haven't considered it for Hauptwerk user configuration.

Thoughts?
Mark S

Interesting idea!

How does it sound if you don't bind threads at all and let Windows distribute the load?

If you can't also redirect audio interface interrupts and DPCs to other cores than the lowest numbered ones that every other activity wants to use, you might still get HW's CPU meter appearing happy when there are buffer underruns in the driver making noise.

And if hyperthreads are enabled, those odd numbered cores may be HT cores. I'd check to make sure you're assigning HW to real cores. Don't know what CPU model you have, so maybe it doesn't support HT.

Glad you like it!

This HW system (and also the other lower performance quad core HW system) has been set to bind threads (with CPU spikes in the green bars). On this HW system, I just tried unbinding. Within 2 measures of a toccata with a full plenum, the CPU spikes on the HW meter were significantly worse (in the red bars repeatedly, but surprisingly without audio drop-outs or glitches) and stayed worse throughout the piece. So it appears better to continue to bind threads.

About using hyperthreads or not - previously, the Cantabile - GlitchFree guide on p. 58 recommended turning off hyperthreading to improve real-time audio processing. I have had hyperthreads disabled for a long time. With 28 physical cores, I have been able to afford to keep hyperthreading off - to dedicate physical cores to corresponding single threads.

About odd cores being HT cores and possibly Set Affinity to odd cores being worse than even cores - I just compared for 30 minutes. To my surprise Set Affinity to 8 high-numbered even cores appeared to have the same range of CPU spikes in the green bars and no audio drop-outs or glitches as the previous Set Affinity to high-numbered odd cores. Can't explain this result, but it could give me more Set Affinity core choices to explore.

Across 30 minutes of playing fast and furiously, with bind threads / hyperthreading off / set affinity on 8 high-numbered cores, there was only 1 CPU spike into the red bars and no drop-outs or glitches which I could hear.

Of course, I need to play several hours more to collect enough data on how much improvement this really is or not.

It is not a comprehensive solution at all (I dream of dedicating cores to exclusively run Hauptwerk only). But for the couple of hours of playing so far, 1 or couple of drop-outs or glitches per hour has been quite the improvement over 1 to many drop-outs or glitches per minute.

Mark S

With HT off, the odd cores are physical cores.

Oh silly me, of course, all are physical cores. Clearly I am having to spend too much time away from OS and computer architecture to avoid stumbling.

Thanks

[mdyde]

Hello Mark S,

Does LatencyMon ( https://resplendence.com/latencymon ) report any problems when the audio glitches occur? E.g. if a driver/interrupt runs for a too long a time on a core on which a Hauptwerk audio engine thread is running (regardless of whether the thread was put there by Hauptwerk or Windows), then an audio glitch is very likely. Unless that's resolved, you're unlikely ever to have reliable audio on the PC -- it isn't realistically feasible to force the affinity manually of every driver and every process that might run.

In Hauptwerk, un-ticking the "Bind audio engine threads to CPU cores on Windows" general preference allows Windows to move Hauptwerk's audio engine threads around as it sees fit, which may help in cases where some other very high priority process (or driver) is hogging the resources on a particular core. However, if there are underlying driver/DPC problems then you still need to resolve them otherwise you will get audio glitches.

Re. the Windows "CPU Sets" API: I don't know off-hand whether that would perform any better/worse, but key parts of it require Windows 11+ (and/or recent Windows Server versions, which almost no Hauptwerk users would use), and we still need Hauptwerk to work on Windows 7 (and most definitely 10), which a lot of Hauptwerk users still use. I've logged an enhancement request to investigate it in the future. However, it almost certainly still wouldn't help if the PC has underlying driver/DPC latency problems, because it would leave Windows free to move Hauptwerk cores around any physical cores, and Windows probably still wouldn't know about cores that had problematic drivers running on them. Hence I doubt it would gain much compared to simply having "Bind ..." un-ticked.

I would suggest;

- Make sure that any problems that LatencyMon reports are fully resolved.

- If that still doesn't solve it, try keeping the "Bind audio engine threads to CPU cores on Windows" general preference unticked (and preferably also keeping "Try to run Hauptwerk at real-time priority on Windows" ticked, and launching Hauptwerk 'as administrator).

- If that still doesn't solve it (and keeping "Bind ..." un-ticked), as a test, when Hauptwerk isn't running, you could could conceivably try editing Hauptwerk's Config.Config_Hauptwerk_xml in a text editor to set AudioEngine_MaxNumberOfCPUCoresToUse to a non-zero value that's more than 8 but less than the number of cores that the PC has. In that case, if set to N, Hauptwerk would only create N-4 audio engine threads. (If left at its default of zero on a computer with more than 8 logical CPU cores, Hauptwerk would create M-4 audio engine threads, where M is the number of logical cores that the computer has.) With less audio engine threads, maybe you'd be lucky and Windows then would happen not to put them on a core that has something problematic running on it. However, it shouldn't be necessary because (provided no driver/DPC latency issues, and "Bind ..." not ticked) Windows should be moving the threads to avoid such cores anyway. [Caveat: editing Hauptwerk's settings files is unsupported, at your own risk, could easily break/crash Hauptwerk, and is actually against the licence agreement unless we've specifically advised someone to do it.]

I just thought I would mention the opposite approach idea, in case Martin hasn't considered this for a Hauptwerk implementation enhancement or other Hauptwerk PC users haven't considered it for Hauptwerk user configuration.

I did briefly consider adding a user option for advanced users to specify a desired process affinity manually, but quickly rejected that because very few people would have the level of technical knowledge to understand or use it properly, and it still wouldn't help unless the user set also the affinity of every potentially-problematic driver and process manually when they launched (e.g. via scripts).