CPU Spikes to Red Causing Intermittent Audio Artifacts

Thanks, abaymajr.

After a few days of testing, Hauptwerk 7.0.1.005 is absolutely stable, rising the use of CPU cores according to demand in a distributed and homogeneous way.

Just to clarify, that isn't a standard Hauptwerk v7.0.1.005 build -- it's a special executable that has all thread-affinity temporarily commented out (so that affinity is never set), solely meant to be sent to people as a work-around of last resort for PCs for which it isn't possible to stop some other process hogging a specific CPU core that Hauptwerk would normally use.

But I was curious to know what kind of application or service would be disputing with Hauptwerk v7.0.0.136 a single core to the point of making it stable at 92% load, even with Hauptwerk in idle, so I could optimize this Hauptwerk setup even further. Do you know any software that could perform this diagnosis?

Unfortunately I'm not aware of any way within Windows to query a process's thread affinities, so probably the only way to do it would be to look at the processes on the Details tab in Task Manager, paying particular attention to any that have a higher-than-normal base priority, or are running with elevated privileges, and are using quite a bit of CPU time (sporadically or constantly), then try right-clicking on them, selecting "Set affinity" then just clicking OK (which evidently causes Windows to reset their thread affinities back to those of the process, i.e. 'all cores' by default), until you find the process that's causing the problem.

I hope I'm not extending this discussion beyond what it would be productive for everyone but, investigating a little further into the matter under my personal circumstances, I couldn't find any process (application or service), other than Hautpwerk.exe, that would cause the >92% single-core utilization of a CPU with 13 other cores. I manually closed all applications and services and none of the shutdowns made any difference in the reported phenomenon. The only process shutdown that changes that core's usage is Hauptwerk.exe itself. Running it in real-time or at normal priority doesn't make any difference either. Changing the affinity of this process for any other group of cores with very low or zero utilization transfers to these the very same phenomenon of overutilization of the previous core. So, there really seems to be some peculiar optimization to be done with that exact processor (Intel i7-13700H) or with its family/generation. Fortunately, as already reported, after replacing Hauptwerk.exe executable with its v7.0.1.005 counterpart, everything is solved immediately and automatically. According to some additional tests I have done today, in this release the normal priority produces the same glitch-free audio performance and no CPU spikes in the HW7 metric as with the priority set to real-time, with an interesting advantage of achieving a higher loading performance (of compressed working files) between 630 and 650 MB/s (around 5% performance gain, compared to real-time priority).

[mdyde]

Hello abaymajr,

It wouldn't necessarily be the case that whatever process (or processes) are normally preventing Hauptwerk getting the CPU time it needs on one or more cores are obviously using very high CPU core utilisation (e.g. >80%) as reported by Task Manager. Also, if restarting a process it will probably set its thread affinities as before again, so that probably wouldn't be expected to help. A brute-force approach might perhaps be to write a script that sets process CPU affinities to 'all cores' (assuming that does the same thing as simply opening and OKing a process's affinity window from Task Manager, i.e. resets all of the process's threads' affinities back to 'all cores') for each running process except Hauptwerk.

Anyway, if that executable solves the problem on your PC, then that's fine, so it probably isn't worth worrying too much about for now.

[mdyde]

P.S. If running Hauptwerk at real-time priority (launched as Administrator, etc.) then you're unlikely to get problems from Hauptwerk not being allowed to set its own thread affinities (e.g. by using that special executable) anyway, since, in my experience, for real-time processes specifically, although they will still often be assigned to inappropriate cores initially, Windows moves threads to more-appropriate cores essentially instantly when needed, without causing an audio glitch. (However, non-real-time processes' threads often don't get moved until it's too late -- after an audio glitch has resulted -- and when Windows does eventually move them it's likely also to cause an audio glitch.)

P.S. If running Hauptwerk at real-time priority (launched as Administrator, etc.) then you're unlikely to get problems from Hauptwerk not being allowed to set its own thread affinities (e.g. by using that special executable) anyway, since, in my experience, for real-time processes specifically, although they will still often be assigned to inappropriate cores initially, Windows moves threads to more-appropriate cores essentially instantly when needed, without causing an audio glitch. (However, non-real-time processes' threads often don't get moved until it's too late -- after an audio glitch has resulted -- and when Windows does eventually move them it's likely also to cause an audio glitch.)

Thanks for those information. To be safe, I'll leave the alternative executable at real-time priority then.

[mdyde]

Thanks, abaymajr. Yes -- I'd recommend that.

I'd just like to update this thread regarding the improvements provided by the recently released HW8. The phenomenon of ">92% single-core utilization of a CPU" on my i7-13700H-based MiniPC seems definitely solved, even turning the new feature "Bind audio engine threads to CPU cores on Windows" on. With only the organ blower sounding, there is still a higher percentual usage of one of the processor cores, but it's way lower than that 92% and the distribution of overall CPU usage among the other cores is close to what happens with the same feature turned off, including the E-cores cores, which were almost completely idle in HW7 without the alternate executable.

[mdyde]

Thanks, abaymajr.

Excellent. I'd recommend keeping "Bind audio engine threads to CPU cores on Windows" ticked (as it is by default) unless one really has to turn it off (due to some other process causing high CPU usage on one of the cores that Hauptwerk is trying to use, if specifically resulting in audio glitches). If turning that preference off, having the "Try to run Hauptwerk at real-time priority on Windows" preference ticked (and launching it 'as Administrator') would probably be important.

Thanks, abaymajr.

Excellent. I'd recommend keeping "Bind audio engine threads to CPU cores on Windows" ticked (as it is by default) unless one really has to turn it off (due to some other process causing high CPU usage on one of the cores that Hauptwerk is trying to use, if specifically resulting in audio glitches). If turning that preference off, having the "Try to run Hauptwerk at real-time priority on Windows" preference ticked (and launching it 'as Administrator') would probably be important.

Now, at least in my strict and concrete case, there is no longer any need to not bind HW engine threads to CPU cores. As for running Hauptwerk.exe process at real-time priority, I'll maintain that strategy due to the absolute glitch-free experience I've had with it for many years.

[mdyde]

Thanks, abaymajr.

I am so glad to have found this discussion.
I have exactly the same problem: sudden CPU usage for a second and sound glitches - even if it's a mini sample set with a CPU reading of usually at 5% max.
With Hauptwerk 8, I set out once again to solve the problem, because I finally want to switch from my (working but slow) Mac mini to the powerful PC.
- AMD Ryzen 9 4900H 3.3 GHz
- 8 core, 16 logical processors
- 64 GB RAM

If I understood this discussion correctly, the cause is probably some process / driver that suddenly claims one CPU core.

Which definitely solved the problem (each on its own):
- The trick with opening and closing the Windows processor affinity screen for Hauptwerk.
- Disabling the new setting "Bind audio engine threads to CPU cores on Windows“
- Enable the setting "Try to run Hauptwerk at real-time priority on Windows" and start as administrator

Which does NOT solve the problem:
- set the affinity of the window manager (dwm.exe) manually to CPU0-3
- all other attempts to kill any processes or force them to CPU0-3.

Please correct me if I am wrong, but my assumption is: Some processes briefly load a certain CPU core and if Hauptwerk has them permanently assigned, there may be audio glitches. Unchecking the "Bind" option lets Windows take over this distribution, but this can lead to performance losses if the CPU is heavily loaded. The "Real-time" option gives Hauptwerk a higher priority than (almost) all other processes, which greatly reduces the risk of audio glitches.

Of course, I'd still like to find out the culprit at some point, but I'm glad to have found ways to run Hauptwerk on the powerful computer and enjoy higher quality settings and larger sample sets.

[mdyde]

Hello Lars,

Please correct me if I am wrong, but my assumption is: Some processes briefly load a certain CPU core and if Hauptwerk has them permanently assigned, there may be audio glitches. Unchecking the "Bind" option lets Windows take over this distribution, but this can lead to performance losses if the CPU is heavily loaded. The "Real-time" option gives Hauptwerk a higher priority than (almost) all other processes, which greatly reduces the risk of audio glitches.

Yes -- I think that's probably a fair summary. Unless you subsequently manage to identify the culprit process, I'd recommend keeping the 'real-time' option ticked (and launching as administrator), and then (only) if that still doesn't completely eliminate the problem, consider also unticking "Bind audio engine threads to CPU cores on Windows“.

If you haven't already done so, one other thing to try would be to run Resplendence' LatencyMon ( https://resplendence.com/latencymon ) temporarily, and see whether that reports any problems when the glitches occur. E.g. perhaps the PC has a driver or hardware performance issue somewhere.

Not sure if this is relevant but it gave me a shock!

I have just built a new Hauptwerk PC. It is not fully up and running yet so I am playing with it. Yesterday, while all was quiet, my new PC suddenly sprang into life, the fans all came on and the CPU monitor shot up to 99%. Oh my goodness, it is melting down, I thought.

It quickly stopped and I traced the incident to the "Windows System Assessment Tool" that, somehow had got itself installed via the Windows 11 Task Scheduler and was running itself periodically. The cheek! It seems that it does not normally make its results known to the users so who is it doing it for, then, I wondered?

However, a few deft applications of the Delete key and it is now not in Task Scheduler any more.

My Hardware DATA:
- AMD Ryzen 9 - 7950X - 16-core, 4.5GHz
- 128Gb RAM
- All 3 x Samsung SSD - including 2 x 2Tb M.2 - one with Windows 11 and Hauptwerk 8 installed
- MOTU - UltraLite AvB, 24Ao (connected by Ethernet), and micro lite — 5x5 USB MIDI interface
- 3 x 3 stereo speaker pairs, Group A --> C; one stereo Rear Channel, and one stereo MID Channel (All Adam A7V)
- 1 Sub MONO
- Hybrid setup with 37 ranks of real pipes (original console only 3 manuals --> New 4-manual console resultant)

SUCCESSES:
- Got all speaker Groups A --> C working, cycling through properly - thanks to excellent Hauptwerk documentation and a few YouTube videos. The mixer was hard-going and only now mainly makes sense to me.
- All Adam A7V speakers connected via ethernet - allowing incredible software control - including turning on/off individually and remotely and precise frequency response tailoring, and TROUBLESHOOTING (all the wiring notwithstanding). This is very beneficial for some of the speaker group placements placed 30+ metres away.
- Hauptwerk 8 noticeable faster at loading and unloading huge sample sets (but didn't stop audio artifacts).
- Whole organ sounds incredible with never a problem with all the MIDI outputs to all pipe ranks, 3 x LCD meters as well as so many MIDI inputs too - the usual 4 manuals, pedal, 58 thumb pistons, 4 swell shoes, etc.

SMALL QUESTION:
- Some of the LED representations on the touch screen menus - Mixer, Bus, and Rank Routing screens - easily understood and are VERY helpful; some of the red LEDs have 3 states - off, dark red, and bright red. What is the difference between the different kinds of dark red and bright red? (for interest's sake)?

MAJOR QUESTIONS:
- For years I've had major audio artifacts; most times, if left for a long enough time, CPU usage would climb down from approaching or penetrating RED to 3 or 4 green bars, but the process took at least 30+ minutes. Suddenly saw a YouTube video addressing this very issue and, when invoked, CPU went to MAXIMUM of 3 green bars - most times 1 green bar. Once, during a vulnerable/exposed moment, the glitch stopped all audio everywhere but, mercifully, spared the MIDI and the pipes continued unaware; but that was incredibly audible! I had to restart MIDI to recover the audio (awkward).

However, as detailed in a few of the recent above posts, the procedure has to be invoked every time the organ is started or there is any appreciable change in the Audio setting. I was so thankful that this seems/seemed to cure the vexing issue - and I can open and close the processor affinity screen - changing nothing - under task manager - in a few seconds. One would presuppose that a batch file could be written (heavens, in DOS) that would open Hauptwerk in administrator mode and do the flash dance at each invocation of this incredible Hauptwerk program - although I haven't seen it. I understand - from a post just a few previous to this - that there are other methods, but I have not tried any of them other than the "affinity method." I did notice looking at the CPU processes involved, that THREAD 7 (of 32) gets hammered relentlessly (100%) the moment Hauptwerk goes live.

- My "only" remaining problem is that I can't seem to get the mixer (or mixer bus, or Rank Routing) to give me a mixdown of all the FRONT speakers. The moment I start changing any of the working groups outlined above, I destroy them - often losing the SUB in the process. This should be the easiest part but I can't seem yet to be successful in that. I am using the Rotterdam sample set at the moment and want to get that completely working before moving on.

Respectfully,
RWAbacus

Mixdowns are sent from the primary buses (of your front speaker pairs for a front mixdown) to either a master bus or an intermediate bus. Groups are not involved, so I don't know how making a mixdown could mess up your groups.

In the Audio Mixer window (not audio Mixer Bus Groups), with the front primary buses selected, check the box for the master or intermediate bus you want to become the mixdown bus. Then select that mixdown bus and give it an output pair of channels, add reverb, set it to record, whatever you wanted it to do.

About the need to set affinity after each organ load, does that still happen if you turn off the General settings | Advanced option that binds threads to cores? I ask because I had read that the cause was a failure of the system call to set threads' or thread groups' processor affinities. [CORRECTION: This facebook rumor about why this happens was wrong. See Martin's explanation below.] If HW doesn't have to bind threads, *maybe* it skips those failing calls at each load, and setting affinity once on the start command could solve it. I can't test this because the problem doesn't happen on my machine, so it's just speculation. [Not binding threads vs binding threads and then manually unbinding them gives the same result. Duh.]

[mdyde]

Hello RWAbacus,

To add to Mark's reply:

SMALL QUESTION:
- Some of the LED representations on the touch screen menus - Mixer, Bus, and Rank Routing screens - easily understood and are VERY helpful; some of the red LEDs have 3 states - off, dark red, and bright red. What is the difference between the different kinds of dark red and bright red? (for interest's sake)?

The green, blue and red virtual LEDs down the left-hand side of the real-time audio and MIDI settings screens (Mixer, etc.) only actually have two states -- on (lit) and off (not lit). However, if you have one or more rows selected in the list then those whole rows may be highlighted a little (in pale blue on Windows 10) to indicate that they're selected, which slightly affects the colour of their virtual LEDs.

- For years I've had major audio artifacts; most times, if left for a long enough time, CPU usage would climb down from approaching or penetrating RED to 3 or 4 green bars, but the process took at least 30+ minutes. Suddenly saw a YouTube video addressing this very issue and, when invoked, CPU went to MAXIMUM of 3 green bars - most times 1 green bar. Once, during a vulnerable/exposed moment, the glitch stopped all audio everywhere but, mercifully, spared the MIDI and the pipes continued unaware; but that was incredibly audible! I had to restart MIDI to recover the audio (awkward).

However, as detailed in a few of the recent above posts, the procedure has to be invoked every time the organ is started or there is any appreciable change in the Audio setting. I was so thankful that this seems/seemed to cure the vexing issue - and I can open and close the processor affinity screen - changing nothing - under task manager - in a few seconds. One would presuppose that a batch file could be written (heavens, in DOS) that would open Hauptwerk in administrator mode and do the flash dance at each invocation of this incredible Hauptwerk program - although I haven't seen it. I understand - from a post just a few previous to this - that there are other methods, but I have not tried any of them other than the "affinity method." I did notice looking at the CPU processes involved, that THREAD 7 (of 32) gets hammered relentlessly (100%) the moment Hauptwerk goes live.

If the method described in that YouTube video (right-clicking on the Hauptwerk process to set its CPU affinity, then clicking OK) resolves your audio glitches problem then it will be because some very high-priority thread belonging to a non-Hauptwerk process (e.g. a driver component) is bound to a particular CPU core which Hauptwerk is also using for its audio engine, resulting in Hauptwerk's audio engine thread being 'choked' and an audio glitch.

If possible, the best way to solve that is to identify the relevant process and avoid having it running at the same time as Hauptwerk, or to persuade it not to bind to any specific CPU core.

However, if you can't easily do that, then in Hauptwerk v8+ there's no need to use the work-around described in that video; instead, simply un-tick the "General preferences | Advanced | Bind audio engine threads to CPU cores on Windows" option in Haupwterk, which achieves the same result. If that option isn't ticked, Windows will move Hauptwerk's threads around amongst CPU cores as and whenever it sees fit. However, if turning that preference off, also having the "Try to run Hauptwerk at real-time priority on Windows" preference ticked (and always launching Hauptwerk 'as Administrator') would probably be important, because in my experience, if not running at real-time priority, Windows often puts audio engine threads on inappropriate CPU cores initially, and an audio glitch occurs when it subsequently moves them (e.g. when you come to play the first large chord).

- My "only" remaining problem is that I can't seem to get the mixer (or mixer bus, or Rank Routing) to give me a mixdown of all the FRONT speakers. The moment I start changing any of the working groups outlined above, I destroy them - often losing the SUB in the process. This should be the easiest part but I can't seem yet to be successful in that. I am using the Rotterdam sample set at the moment and want to get that completely working before moving on.

As Mark mentioned, you don't need to do anything involving ranks, rank routing, or groups. Everything needed for configuring mix-downs is on the "General settings | Audio mixer" screen. Ranks are routed (directly, or via groups) to 'primary' mixer buses, and the 'master mix' and 'intermediate mix' mixer buses are for mix-downs from the 'primary' mixer buses.

For example, on the "General settings | Audio mixer" screen:

- First, in the left-hand list of buses (within the node for the audio mixer preset that the organ is set to use), highlight just "Intermediate mix 1".

- Now in the right-hand pane, give it an appropriate name, such as "Front mix-down", and select the desired audio device channel(s) for it.

- Now, in the left-hand list of buses, highlight all of the *primary* mixer buses that you're using for your front speakers (and no others).

- Then, in the "Sends: to intermediate mix buses" section at the bottom of the right-hand pane (you may need to drag the screen taller to see them), make sure that intermediate mix bus 1 is ticked. That ensures that those primary buses will also send audio to intermediate mix 1 (your "Front mix-down").

[Also, with just "Intermediate mix 1" highlighted in the left-hand bus list, you may want to un-tick all of its "Sends to master mix buses", otherwise the signal from you front speakers would be reaching your sub-woofer (for example) from two places, thus doubling its amplitude, which you probably wouldn't want.]