Errors: 'Corrupted data. Resetting data reciever.' and 'Bad payload legth'.

[nathanaeloneill]

Hi Kevan,

We have SCT/ODR/Driver setup in the following manner:
1x laptop is connected to 1x network switch. To this switch is connected 2 LWDAQ drivers, and each driver is connected to an ODR. The IP addresses of the two LWDAQ drivers are 10.0.0.37 and 10.0.0.38.

This set-up has run for many months without issue. However, we are now getting some persistent error messages. Errors occur more on the 10.0.0.38 LWDAQ driver and are as follows:

ERROR: Corrupted data. Resetting data receiver.
and
ERROR: Bad payload length "0", try "16" instead. Resetting data receiver.

Do you have an idea of the probable cause of these two errors?
I compared the configured settings of this system with a system that is currently working well. The only difference I could see is that on the system that is working without error, messages_per_clock is 1. On the system with errors there are 5 messages_per_clock.

What does 'messages_per_clock' correspond to? Do you think this could be a/the cause of the error?

Many thanks

Nat

[Kevan Hashemi]

Dear Nat,

Apologies for delay in answering your post. Not sure how we missed it.

This corrupted data error and the wrong payload error both arise from loss of TCPIP packets coming from the LWDAQ Driver. We see loss of TCPIP packets when our data acquisition (DAQ) computer starts using the wireless network to contact our telemetry server (any of LWDAQ Driver, Animal Location Tracker, or Telemetry Control Box). We see these errors when our DAQ computer uses the wireless network to contact some external server, such as a time synchronization service, and then forgets to switch back to the wired network when trying to contact the telemetry server.

The "bad payload length" is an old error message. The latest LWDAQ handles these corruption errors more effectively and provides better error messages. If you can update both machines to LWDAQ 10.6.7, which we just pushed today, that would be great.

Is your DAQ computer's wireless network active? Do you have two wired networks attached to your computer?

Best Wishes, Kevan

[nathanaeloneill]

Hi Kevan,

Sorry for the radio silence -- the solution of updating LWDAQ worked well.

We now have a similar but different error:

ERROR: Data corrupted, timestamp errors found by analysis. (24-Jun-2024 13:16:35)
WARNING: Corrupted data, resetting receiver. (24-Jun-2024 13:16:35)
WARNING: Download resumed after interruption of 3 s. (24-Jun-2024 13:16:38)
ERROR: Data corrupted, timestamp errors found by analysis. (24-Jun-2024 13:23:22)
WARNING: Corrupted data, resetting receiver. (24-Jun-2024 13:23:22)
WARNING: Download resumed after interruption of 3 s. (24-Jun-2024 13:23:25)
ERROR: Data corrupted, timestamp errors found by analysis. (24-Jun-2024 13:30:20)

This is now quite regular, but I'm unsure what is causing it. This error is only observed on one of our set-ups, the only difference i can see between set-ups is that the ethernet to USB adaptor is older. Is this an error that can be caused by old, or dying usb to ethernet adapters? do you think we should replace it. I should also say that this issue is on a set-up that currently has a high recording load: with the maximum number of transmitters recording and lots of video (microseven to PoE switch to ethernet-usb adapter).

Thanks, as always,

Nat

[Kevan Hashemi]

Dear Nathan,

I am able to reproduce the same exact errors in the following way, using LWDAQ 10.6.10. I start two Neurorecorders and have them download from the same Octal Data Receiver. At first I get only the following warnings in both Neurorecorders:

WARNING: Encountered 3 errors in received interval. (24-Jun-2024 10:02:38)

But if I open the Receiver Instrument using the Receiver button in one of the Neurorecorders, I start to get these errors in the Neurorecorder:

ERROR: Data corrupted, timestamp errors found by analysis. (24-Jun-2024 10:02:33)
WARNING: Corrupted data, resetting receiver. (24-Jun-2024 10:02:33)
WARNING: Download resumed after interruption of 4 s. (24-Jun-2024 10:02:37)
ERROR: Data corrupted, timestamp errors found by analysis. (24-Jun-2024 10:02:39)
WARNING: Corrupted data, resetting receiver. (24-Jun-2024 10:02:39)
WARNING: Download resumed after interruption of 1 s. (24-Jun-2024 10:02:40)

Please check to see if you have two Neurorecorders trying to access the same Octal Data Receiver, and if you do, please confirm that the one showing the above errors is associated with an open Receiver Instrument panel. If all is as I describe, close one of the Neurorecorders.

In the long run, I don't like the fact that the Neurorecorder displays different error messages depending upon whether or not the Receiver Instrument window is open, so I'm going to look into why that's the case. If there is no good reason, I will simplify the behavior in LWDAQ 10.6.11.

Best Wishes, Kevan

[nathanaeloneill]

Thank you, Kevan. I will check to see if this is the case and let you know.

[nathanaeloneill]

Hi kevan,

I closed and restarted LWDAQ ensuring that only one neurorecorder was open and called neuroreciever from this i.e not from the instrument menu.

I also deleted a hidden old version of LWDAQ to make sure there was no interference from this.

The errors still persist:
WARNING: Download resumed after interruption of 2 s. (24-Jun-2024 19:49:19)
ERROR: Data corrupted, timestamp errors found by analysis. (24-Jun-2024 19:53:30)
WARNING: Corrupted data, resetting receiver. (24-Jun-2024 19:53:30)
WARNING: Download resumed after interruption of 3 s. (24-Jun-2024 19:53:33)
ERROR: Data corrupted, timestamp errors found by analysis. (24-Jun-2024 19:58:00)
WARNING: Corrupted data, resetting receiver. (24-Jun-2024 19:58:00)
WARNING: Download resumed after interruption of 2 s. (24-Jun-2024 19:58:02)

Could too many transmitters cause this problem?

Thanks

Nat

[Kevan Hashemi]

Dear Nat,

> Could too many transmitters cause this problem?

No. There is another computer accessing that driver. Could you check the other computers in the same room. I believe you have three connected to the same subnet. Each LWDAQ Driver has a different IP address. There must be a Receiver Instrument or Neurorecorder trying to record from the same address.

Best Wishes, Kevan

[nathanaeloneill]

Thanks Kevan, we have 3 laptops and 3 witches and 4 LWDAQ drivers and octals:

Setup 1: 1 laptop--> switch-->LWDAQdriver.10.0.37-->octal
Setup 2: 1 laptop--> switch-->LWDAQdriver 10.0.38-->octal (the one with this problem)
Setup:3 1 laptop-->switch-->2*LWDAQdriver 10.0.37+10.0.38>2*octal

One change recently made us connecting setup3 to the Ethernet port instead of the wifi. Do setup3 you think it is finding driver from setup 2 through the institutes internet? I'll double check the wiring tomorrow to confirm setup 3 hasn't been connected to setup 1's switch.

Thanks a lot!

Nat

[Kevan Hashemi]

Dear Nat,

Thanks for details.

I'm looking at the timing of your errors. I should have done so earlier. They are separated by several minutes. If there are two Neurorecorders accessing the same Octal Data Receiver, they cannot be on the same computer, or else they would conflict all the time. Nor can they be accessing the same Octal Data Receiver, because they would likewise conflict all the time. So it must be that there are two computers accessing two Octal Data Receivers, and these Octal Data Receivers share the same IP address, so that they conflict some of the time.

> Do setup3 you think it is finding driver from setup 2 through the institutes internet?

That's possible if the institute's internet is also using subnet 10.0.0.x, but I doubt that is the case.

> I'll double check the wiring tomorrow to confirm setup 3 hasn't been connected to setup 1's switch.

I think you mean Setup 3 might have been connected to Setup 2 (the one with the problem). If so, I would expect you to see the same problem in Setup 3. Do you see any warnings in the Neurorecorder of Setup 3? How about Setup 1?

I suggest you give the LWDAQ Drivers distinct addresses. You could try 10.0.0.37, 38, 39, and 40, while making sure that the local IP addresses of your three laptops are also distinct, such as 10.0.0.2, 3, and 4. You have a bunch of MicroSeven cameras in there as well, so you would have to make sure you don't conflict with those either, but it seems to me worth the trouble of removing this worry by giving your LWDAQ Drivers distinct IP addresses. To change the Driver IP addresses, use the Configurator.

Thank you for working on this problem. The intermittent data loss really messed with Rob Wyke's recent recordings of cortical spreading depressions.

Best Wishes, Kevan

PS. Another thing I want to note, although I don't think it's particularly relevant. When I was up in the animal house two weeks ago with Rob and Luiz trying to fix this problem in person, I wanted to clear the "router address" for the Ethernet interface of the computer that was skipping data. The router address (or "gateway") was set to 10.0.0.1. We don't want the computer trying to get to the outside world by using 10.0.0.1 as a gateway. But I was unable to delete the router address in that particular version of Windows. So it is possible that this particular machine will try to access the internet through 10.0.0.1. If it did, I don't see how that would cause problems for your telemetry, but it might cause problems for your file sharing over your institute's network.

[nathanaeloneill]

Thanks Kevan, I will try and systematically work through this and resolve it and post when we have it sorted.

I will look for the default gateway as well, I seem to remember this is a modifiable setting on the USB to Ethernet dongle.

Thanks a lot for your help today I'll keep you posted this week

Cheers

Nat