Neurorecorder File Conflict

[Kevan Hashemi]

A customer writes:

"We have an issue with the ALT recording. I think the user has clicked on the recording button too fast between 2 instances. So, the 2 interfaces (window instances) created the same file name, creating conflict and overwriting on each of one. Please find the file to this link <snip>. Can we do something to recover the file ? I don't think so, but I prefer to ask you just in case;-)"

[Kevan Hashemi]

If I understand correctly, you had two Neurorecorder windows open, the user clicked Record in one, moved quickly to the other, clicked record, and the two files began in the same clock second, so they have the same name.

I have always assumed that the user will store the files from each Neurorecorder in a different directory, so this conflict would never occur. But I should not have made that assumption. We will update the Neurorecorder so that it generates a warning if the recording file already exists, then waits for a few seconds, and tries again.

I have your file, M1691072178.ndf. It is 125 MByte, which is about the right size for 6 transmitters running at 256 SPS for an hour (6 * 256 SPS * 20 Byte/Sample * 3600 = 110 MByte). When I open it in the Neuroplayer, in the first second I see six transmitters 185, 186, 187, 188, 189, and 193. I move through the archive 1/16 s at a time, and I see either the first three transmitters or the second three transmitters. Occasionally, there is an interval where the data is corrupted. I am not sure why the corruption occurs. I will have to think about it.

If I can figure out the corruption, then we might be able to recover the data from these files, and create two separate NDF files. Is it just one file that is corrupted in this way, or many files?

[Kevan Hashemi]

Customer replies:

> Is it just one file that is corrupted in this way, or many files?
Yes we have many files.

Kevan says:

Oh dear. We will try to figure out how to separate them.

[Kevan Hashemi]

Dear Greg,

We have an NDF file M1691072178.ndf, size 125 MByte that is the result of two Neurorecorders writing to the same file. Each Neurorecorder is downloading from its own ALT the signals from three 256 SPS SCTs. The recording took place on a Windows machine. Our hope was that the two Neurorecorders would take turns writing to the files, each writing a valid block of data to the file without corrupting the date written by the other Neurorecorder.

But we find that most of the data in the file has been corrupted by over-writing by the two Neurorecorders. The corruption is so severe that we cannot even see individual four-byte SCT message headers in a hexadecimal dump of the corrupted data. The best we can do is extract all intact messages from the SCT channels we know to belong to a particular Neurorecorder. Thus we are able to extract the un-corrupted data.

At the link below is Signal_Separator.tcl, a LWDAQ Toolmaker Script. To run, cut and paste into the LWDAQ Toolmaker execution window and press Execute.



Before execution, make sure the "ids_A" and "ids_B" strings specify the SCT channels you want extracted from the original combined NDF file. These two sets of channels will be written to two new NDF files, Part A and Part B. The new files will have the same name as the original, but be written to directories A and B within the same directory as the original file. The script creates directories A and B if they don't exist.

We run this script on M1691072178.ndf and we get two output files. They are 762 s and 775 s long. In the Neuroplayer Overview, they look like this:

Part_A_Overview.gif (44.78 KiB) Viewed 8799 times

Part_B_Overview.gif (45.53 KiB) Viewed 8799 times

Of the six channels, five look as if they have survived the extraction without corruption. But No193 does not look right: it undergoes a step down of 32768 counts half-way through Part B, which suggests that somehow the top bit of the sixteen-bit samples has been corrupted. Even without the top bit, however, the signal still looks like EEG.

This appears to be the best we can do for now. The script takes a minute to separate each file, but it will do all your files. In the future, we will avoid this problem because the Neurorecorder will raise an error if the file it wants to create has already been created by another Neurorecorder.

Best Wishes, Kevan

[Kevan Hashemi]

Recommendation: Use separate recording directories for each Neurorecorder.

Detail: Neurorecorder 163 will be available in LWDAQ 10.6, which we hope to release in September 2023. The Neurorecorder raises and error when it sees that the file it wants to create has already been created by another process. After the error, Neurorecorder stops recording. We tried to duplicate the conflict between two Neurorecorders by selecting the same recording directory for both, targeting different data receivers, and pressing Start in each Neurorecorder as quickly as we could, one after the other. In twenty attempts, we generated the error once. In practice, the error may not occur immediately, but some days later, when disagreement between the data receiver clocks causes the two Neurorecorders to come into synchronization and attempt to open a file in the same second. There is only one way to avoid these problems: use a separate recording directory for each Neurorecorder.