Audio pops

Once you get a pop, if you could make two separate, uncompressed (!) recordings without any further processing (such as extra reverb) of the same pop, and upload them here, perhaps it's possible to see if there is some recognizable value being written in memory, or that it's just noise or a single corrupted sample. It would help if you could find the perspective that contains the pop, and limit the recording to only that perspective.

RAM corruption that appears after 12 hours doesn't sound like a normal hardware failure, I must say. But that St. Anne (which fits in much less memory) didn't expose the issue does point at a memory problem.

Everyone,

Thank you all for your excellent suggestions. I verified that the amount of wired RAM does indeed change slightly as the pops emerge after some time.

I got quite frustrated this morning because the Hauptwerk Forum site was not accessible when I wanted to reply to Martin's latest post on this topic! The Forum seems to be down about 10% of the time on average - though I am only sampling this a few times each day. It is a persistent problem. It does not matter what browser or even what device I use either.

In the period when it was unavailable this morning, I decided to take a few steps. First, I have removed from my hard disk the largest sample set I own - the PAB Gravissimo. I haven't used it in over a year and it cannot be customized for my console because of clever rank and pipe file renaming! This freed another 45 GM of disk space immediately.

I am about to upgrade my MAC OS to the latest version. I am mostly doing this so as to be able to use the latest Apple advice on reducing swapping. I am currently waiting fr HW to complete a backup before doing this.

I have been putting off purchasing the iLok USB device - because I have excellent Internet and have never worried about it before. However, this step may also be in my future if pops continue.

I had not considered actually examining the digital data in the pop recordings - but this is also a good suggestion. It reminds me to the days I used to have read core dumps (some of you may have to look that up online to see what I am talking about!) in my early days at JPL.

Les

[mdyde]

Thanks, Les.

Hope something there helps.

If wired RAM varies, then I think it's probably a problem with the RAM. Probably not good news, given the cost of RAM these days! To check you'd have to take out some of the RAM chips, reboot the computer, and test the RAM with each configuration, as I recall. If you take out a RAM chip and you don't get any errors, that's the faulty one there.

It is probable that the operating system makes use of additional “wired memory” whilst HW is not being played so changes in the total amount of wired memory being used may not in itself be an indication that there is any change to the amount of wired memory being used by HW.

Does the memory monitor show that the amount of wired memory has changed after some time when the loaded organ still plays without audio pops? If so, this may indicate this is not related to the problem.

Is there a way to see the allocation of wired memory to individual processes? Can you verify that HW is still recorded as owning the memory it acquired and locked into physical memory when the audio pops are heard?

It is unlikely, but a bug in the os could overwrite a part of memory being used by the HW processes. (Long ago,I found such a bug in a commercial operating systems.)

Well, the changes I made yesterday appear to be for naught. Despite updating the operating system to Sonoma (OSC 14), freeing up lots of HW cached organs, and ensuring I have a lot of free space on my hard disk, the pops are back after a day of having the rogan samples sitting In RAM.

The pop appeared on a new pipe - but I expected it would move given all the other changes.

And, I have a new symptom, possibly:

When I tried to reload my sample set to see if the pop went away (as before), HW crashed! This has never happened before. It generated a Mac trouble report, which I can send to anyone who would like to see it (Martin?), The report begins thusly:

-------------------------------------
Translated Report (Full Report Below)
-------------------------------------

Process: Hauptwerk [1338]
Path: /Applications/Hauptwerk Virtual Pipe Organ/Hauptwerk.app/Contents/MacOS/Hauptwerk
Identifier: com.milan-digital-audio.Hauptwerk.Hauptwerk
Version: ???
Code Type: X86-64 (Native)
Parent Process: launchd [1]
User ID: 501

Date/Time: 2024-01-09 13:21:20.4403 -0800
OS Version: macOS 14.2.1 (23C71)
Report Version: 12
Bridge OS Version: 8.2 (21P2057)
Anonymous UUID: 846956A0-033D-AF42-45F6-7C9A1DA84387


Time Awake Since Boot: 96000 seconds

System Integrity Protection: enabled

--------------------- (total file size was 39 kb

I restarted HW, without any problem. I then reloaded my sample set (a customized version of PCC) and the pops were indeed gone. I will let it run for a day and verify the same pipes are corrupted.

However, I am getting a bit frustrated with all this. Until the HW crash today, I would not have even suggested that HW's application memory was being corrupted - only the sample chase in RAM. I do not know how to test for this - despite a reasonable good knowledge of UNIX.

Les

I have posted the error file I received when I tried to relaunch HW the first time. You can view it here:

http://www.nightbloomingjazzmen.com/NBJ%20Pdfs/HEerorrfile.pdf

[mdyde]

Hello Les,

I think that crash is almost certainly simply another symptom of RAM getting corrupted. In this case it happened to be some class data, rather than sample data. I would strongly suspect that you have a RAM board with an erratic fault, and Apple's diagnostics didn't show it because the board happened to be behaving at the moment you ran them. (Some other small possibilities are that your drive has corruption on it, e.g. within some OS files or an area that's being used by the OS for for paging, or that some other process is corrupting RAM, such as a virus scanner.)

Please try:

- Run Apple's memory diagnostics again in the most thorough mode possible. Do likewise with your drive.

- Download and (re-)install the current version of iLok License Manager: https://www.ilok.com/

- Download and (re-)install the current versions of drivers for your audio and MIDI interfaces.

- Download and re-install the current version of Hauptwerk, leaving all options in the installer at their defaults (so that it upgrades/repairs your existing installation):
https://www.hauptwerk.com/osx/
Verify the MD5 checksum before installing: viewtopic.php?f=10&t=20937#p156778

- Reboot.

- I've just deposited 14-day trial iLok licences for the MDA Salisbury sample set to your iLok account. (It's the largest MDA sample set, and we know it's very well-proven, with a bug-free ODF.) Activate those licences in iLok License Manager.

- Launch Hauptwerk via a 'spare' 'Hauptwerk (alt confg N)' configuration (desktop shortcut) .

- Use 'File | Revert all settings to factory defaults' within that configuration to ensure that all of its settings are (still) at their defaults. (Hauptwerk will exit after you revert the settings.)

- Re-launch that same 'Hauptwerk (alt confg N)' configuration, and select the Mac's built-in output for audio output, and the minimum MIDI IN ports necessary in order to play virtual keyboards. (Don't enable any MIDI OUT ports, so as to eliminate any possibility of MIDI feedback, etc.)

- Download all of the parts of the Salisbury sample set: viewtopic.php?f=10&t=17988
Verify their MD5 checksums: viewtopic.php?f=4&t=20931&p=156504#p156502

- Install all of the parts of the sample set.

- Load the Salisbury volume 3 (=full) ODF, with all ranks set to 24-bit uncompressed (so that the sample set uses as much RAM as possible, which will probably be around 24 GB).

- Auto-detect just the virtual manuals (and nothing else).

- Verify a few times that none of the samples exhibit the 'audio pop' problem. (N.B. since it's a multi-release sample set, you would need to trigger the short, medium, and long releases. E.g. you could play glissandos at different speeds across all keys, with all stops and couplers drawn.)

- After a couple of days, see whether any of the samples now exhibit the 'audio pop' problem. Assuming so:

- Reload the organ.

- Verify again that none of the samples exhibit the 'audio pop' problem.

- Use "Engine | Advanced | Stop audio/MIDI".

- Use "Engine | Advanced | Start audio/MIDI".

- Verify again that none of the samples exhibit the 'audio pop' problem.

- Use "Engine | Advanced | Stop audio/MIDI".

- Leave the system untouched for a couple of days.

- Use "Engine | Advanced | Start audio/MIDI".

- See whether any of the samples exhibit the 'audio pop' problem. (If so, that would almost certainly indicate that the RAM had become corrupted by something external to Hauptwerk, since the audio/MIDI drivers, and Hauptwerk's engines wouldn't have been active at all over those couple of days.)

[Assuming your Mac is a 2018 i7 Mac Mini with 64 GB of RAM, I have one of those here too, so could try exactly the same test for good measure, if needed, Mine has macOS 13.]

[mdyde]

P.S. If reinstalling the latest versions of iLok License Manager, Hauptwerk, and the sample set don't solve it, and if you can reproduce the problem with a sample set that uses no more than about 16 GB, an alternative test (as larason2 mentioned) would be to remove 32 GB of your 64 GB of physical RAM, see whether the problem still occurs, then swap it for the other 32 GB if so, to try to determine which of the RAM boards is faulty (if any).

Martin,

First, thank you for what appears to be an extraordinary effort on your part to help!

Although your offer to use Salisbury as a test case makes a lot of sense, I am getting ready for a month-long vacation abroad in a couple weeks, so I would prefer to try this (if still necessary) after I return in March.

I do not believe this problem will be any different for any large sample set. My custom sets are all CODM-generated and I don't see a mechanism by which they would behave any differently than any "straight to ODF" sample set. The fact that I haven't seen this particular "delay popping" reported my other HW users makes me think this is a problem with my computer and not the sample sets. Many users routinely run much large sample sets than I do - not more ranks, but certainly more perspectives, which really adds up the RAM...

I will try some of the other things you suggest in the near term. I ran Apple's memory diagnostics again yesterday - the maximum number of loops. It took about 60 minutes to run this test - so it still will not find memory cells that flip only every 24 hours. Of my 64 GB RAM, the test covered only 49 GB - because I ran it in a normal MAC mode. All of my tested RAM passed. I might retry this in single-user diagnostic mode - but I still think this is a long shot.

I will reload the iLok license manager (this is an easy thing to do).

I will reload the M-audio MIDI drivers (again, it is easy, but unlikely the problem).

I do not use any special audio drivers as I am using the Mac's headphone jack audio. When I updated the operating system a couple days ago, this updated the audio driver...

I will download and reinstall HW as you suggest as well.

I should be able to accomplish all this today.

Les

Les

[mdyde]

Thanks, Les.

Yes -- it almost certainly isn't due to an error in your custom ODF (and if it's solely CODM-generated then it would originally have been fully validated anyway), but installing a different highly-proven organ (when you're back in a few weeks) could help to rule that out once and for all. Also, it could help in case your ODF or any of its samples (or cache) had become corrupted on the drive.

This sounds to me like a hardware problem. Memory is favourite, but not the only possibility. Sadly it is possible to have faults which diagnostics do not find (I once had a big fight with Univac when I diagnosed a memory fault which their software didn't find; as it was a new board it took a lot of effort to convince them to ship a replacement across the Atlantic!).

Paul

Martin,

Things are not going so well today...

I reinstalled the iLok driver - no problems here.

I cannot update my MIDI driver because m-audio has not come out with a version that is compatible with OSX 14! I am choosing to ignore this for the moment because the interface appears to still be working fine.

I downloaded the latest version of HW and went to determine the file's checksum. However, the installation app appears to the Mac as a folder, not a file. It contains a second folder called "Contents" and several more folders within this. I have no idea how to calculate a check sum. I assume this is actually because it was downloaded as some sort of compressed file and the Mac decompressed it before I could examine it. I have to proceed assuming the file is correct.

I copied all my HW user files to a second computer in case something goes awry in the reinstallation. I am about to start this.

Les

[mdyde]

Thanks, Les.

I downloaded the latest version of HW and went to determine the file's checksum. However, the installation app appears to the Mac as a folder, not a file. It contains a second folder called "Contents" and several more folders within this. I have no idea how to calculate a check sum. I assume this is actually because it was downloaded as some sort of compressed file and the Mac decompressed it before I could examine it. I have to proceed assuming the file is correct.

Yes -- if I recall correctly Safari automatically unZIPs files after download (but Firefox doesn't).

I use Safari by default - so that explains the compression issue.

In any case, I reinstalled HW using all the defaults. It took about an hour. After that, I loaded PCC again and everything played correctly - for now. I'll let you know tomorrow if this made any difference. If not, I think the next step for me is to delete all my organ caches and start over loading the ones I use most often.

Les