Convert multiple files within one folder

[Laurent]

Dear all,

I’m using the LWDAQ software and need to convert > 1000 files from ndf to edf format. I see that LWDAQ can do this, but I cannot find a way to just enter the directory and let LWDAQ process each file successively. Is this possible?

Thanks for your help,
Best

[Kevan Hashemi]

Dear Laurent,

Yes, you can export all files in a directory automatically. See "repetitions" in the Exporter manual:

https://www.opensourceinstruments.com/Electronics/A3018/Neuroplayer.html#Exporting%20Data

You can export to one big file, or many smaller files. I will summarize the procedure for you here.

In the Neuroplayer, select the directory containing your NDF files. The Neuroplayer's "directory tree" is now the directory you selected as well as all subdirectories. The Neuroplayer will be searching the directory tree for NDF files. Press First to make sure you have selected the first file in this directory.

Open The Exporter. Define your export. Suppose you have 1000 one-hour NDF files and you want to export to 24-hour EDF files. Set the duration to 24*60*60, press "Archive Beginning". In the "Repetitions" box, enter "*". After every export, the Exporter will start another export. Please try this and let me know if it works.

Before you proceed with exporting thousands of files, I recommend that you wait one more day until I have released Neuroplayer 171 with LWDAQ 10.6.11. Earlier Neuroplayers failed to calculate the start time of repeated exports correctly. The result is overlap between export files. I am currently working with another group to fix this problem.

I will let you know when Neuroplayer 171 is ready. I would like it very much if you could try Neuroplayer 171 on your data and let me know if it behaves well.

Best Wishes, Kevan

[Laurent]

Dear Kevan,

Thank you for your prompt answer.
I followed your explanations step-by-step. However, the exported files do not have the same name as the original ones. Actually the first file is correct (M1717677796 becomes E1717677796) but then there are slight changes (e.g., the second file M1717681397 becomes E1717681395). Note that I set the duration to 3600 since I want to export the entire length of my files.

If alternatively you have some lines of codes that simply convert the entirety of .ndf into .edf, I could maybe run a .tcl script (if that is easier) ?

Thanks again for your help,
Best
Laurent

[Kevan Hashemi]

Dear Laurent,

> However, the exported files do not have the same name as the original ones.

Indeed, they do not. This is exactly the problem that we are working on in Neuroplayer 171. If you look at the earlier topic on this message board, "reading NDF files in python", you will see Raphael explaining how the files don't have the same names AND they contain overlapping data. The beginning of one file is a copy of the last few seconds at the end of the previous file. This overlap is a serious problem, and that is what we are fixing in Neuroplayewr 171. In addition to fixing that serious problem, we have set up the Exporter so that, when the export file is supposed to have the same length as the original NDF file, the export file will always have the same name as the NDF file.

I am trying to break Neuroarchiver 171 now with Exporter tests. When I am convinced that I cannot break it, I will release LWDAQ 10.6.11 with Neuroarchiver 171 and ask you to test it.

Best Wishes, Kevan

[Laurent]

Ok sure, I'll wait for the new version to be released. Thanks!
Best,
LS

[Kevan Hashemi]

Dear Laurent,

We just released LWDAQ 10.6.11 with Neuroplayer 171 and Neurorecorder 168. You can pull from our Git repository:

https://github.com/OSI-INC/LWDAQ

Or download and unzip our new archive:

http://www.bndhep.net/Software/Download/LWDAQ_10.6.11.zip

Please try the Exporter in the Neuroplayer 171 and let me know if it assigns the correct names to the export files.

When you make more recordings, please note that the default behavior of the Neurorecorder 168 is "free-running" rather than "re-synchronizing". The Synchronization box will be un-checked. We discuss the two synchronization strategies here:

https://www.opensourceinstruments.com/Electronics/A3018/Neurorecorder.html#Synchronization

If you use the free-running strategy, your one-hour NDF files will have names separated by 3600 s, except for occasional differences of 3599 s or 3601 s as the computer and telemetry receiver clocks drift apart. With the free-running strategy, you will not lose one second of data every hour.

Best Wishes, Kevan

[Laurent]

Dear Kevan,

Thanks a lot for for the update. It seems to work, but quite slowly. Based on a few files, I calculated that it would takes days, if not weeks, to convert all my data. Is there any chance that you would have a few lines of code to open ndf files and metadata in Matlab? That would be extremely useful.

Thanks,
Best
Laurent

[Kevan Hashemi]

Dear Laurent,

> It seems to work, but quite slowly.

I have an NDF file containing 14 telemetry signals, each at 256 SPS. I make a 128-s export of all channels to a single EDF file. I start with a 1-s playback interval, with the Value vs. Time (VT) and Amplitude vs. Frequency (AF) plots enabled. The export takes 19 s. Now I increase the playback interval to 8 s. The export takes 5 s. Now I disable the VT and AF plots. The export takes 4 s.

What playback interval are you using? Do you have the VT and AF plots enabled?

With the correct playback interval, you should be able to export 14 channels of 256 SPS at a rate of one one-hour archive every two minutes. You should be able to export 4 weeks of recordings from 14 animals in less than one day. I'm running on a 1-GHz MacBook Air. On a 3-GHz desktop PC, I expect the export to be twice as fast.

> Is there any chance that you would have a few lines of code to open ndf files and metadata in Matlab?

Reliable reconstruction of the signal, where we reject interference messages, fill in missing messages, and remove glitches, takes hundreds of lines of code. I don't have Matlab code to do the job. My hope is that you can get the exporter to run fast enough.

Best Wishes, Kevan