Do Not Play Current Video File: Do not attempt to play the video file that is being prepared on disk by the Videoarchiver. The Videoarchiver deletes this file periodically. If you try to play the file, the Videoarchiver will encounter a file access error and recording will stall. Use the View button to view the incoming video.
Note: This manual applies to Videoarchiver 32+. See Changes for new features if you are using older versions.
[09-JUN-22] The Videoarchiver records video from our Animal Cage Cameras (ACC). The Videoarchiver will also dim and brighten camera lights on a daily schedule. Run the Videoarchiver from the Spawn menu of our LWDAQ Software. When we run the Videoarchiver and the Neurorecorder on the same computer, they record and ACC video and Subcutaneous Transmitter (SCT) biometric signals simultaneously and synchronously. The Videoarchiver allows us to view live video from ACCs, to change their IP addresses, and to turn on and off each camera's white and infrared illumination. The Videoarchiver makes sure that the clocks on the cameras remain synchronous to within ±50 ms of the clock of the data acquisition computer. The Neurorecorder does the same thing for its own recordings. The result is continuous video and biometric recordings synchronized to within ±50 ms. The Neuroplayer allows us to view synchronous video and biometric signals simultaneously, to jump to random locations in continuous recordings, and to export simultaneous video and biometric signals for segements of continuous recordings.
We use a live video display when we adjust the picture acquired by each camera. Each camera has its own View button to display what it sees. When we are not recording from a camera, the view is an un-compressed stream that shows up on your computer screen. The state of the camera will be shown as "Live". The delay between movement in the field of view of the camera movement will be barely noticeable to the human eye. We use the live display to determine the effect of illumination, adjust the focus of the camera, and make sure we have no distracting reflections in the field of view. Once we are satisfied with the picture obtained from the camera, we can start recording. We start recording with the Record button. Our live display will close and recording to disk will start up.
During recording, the un-compressed video stream is not downloaded by the data acuisition computer. Instead, the stream is stored temporarily within the camera, broken into consecutive segments of length one or two seconds. Each segment is namaed after its start time in Unix seconds. The camera compresses these segments, reducing their size by a factor of ten with no appreciable loss in image quality. It is these one-second compressed segments that the Videoarchiver downloads to the data acquisition computer. The delay between movement in the field of view and the arrival of compressed video in the data acquisition computer is the recording lag. The Videoarchiver displays the recording lag in units of seconds on the right side of each camera's control bar. If we press the View button during recording, the Videoarchiver will display the incoming compressed segments. The display will be delayed by recording lag, and the display will flicker every second as a new segment is loaded into the video player. If we move or change the size of the recording view, the display resets its position and size every second. The recording closes itself automatically after monitor_longevity seconds. If we leave the monitor view open, it tends to freeze after ten or twenty minutes. Just press View to open it again.
We can change the size of our live video display with the display_zoom parameter. By default, this is 1.0, but we can set it to 0.5 to make a half-size display. So long as we are not recording with any of our cameras, we can open live views for all of them. We press the View button in one camera after another to open a window for each of them. But a camera's live view will cloase as soon as we start recording from the camera. If we are recording from several cameras, we can view only one recording at a time. Press the View button and the recording view will open, or the existing recording view will switch to the new camera.
By default, the Videoarchiver adds new segments to its video recording file once every sixty seconds, and every ten minutes it creates a new recording file. We must refrain from viewing the recorded files why they are being built by the Videoarchvier. Doing so can cause the Videoarchiver to stall. We can turn on and off the white and infrared lamps of individual cameras using their individual control buttons. We can turn off all the lamps with a single press of Lamps_Off. And we can set up a schedule for turning on and off the lamps of all cameras with the Scheduler.
[09-JUN-22] The Videoarchiver is written in TclTk. It is available in the Tool Menu of the LWDAQ Program. Before you can use it, you must download the Videoarchiver libraries, which are available in Videoarchiver.zip. When you open the Videoarchiver for the first time, you will see the following message. Click on the link to download the Videoarchiver software.
Download and decompress the zip file. The result is the Videoarchiver directory. Place the Videoarchiver directory next to your LWDAQ directory. Having followed our Animal Cage Camera set-up instructions, the Videoarchiver should be able to communicate with your camera. The cameras and the data acquistion computer will be connected to the same Power-over-Ethernet (PoE) switch, forming a local Ethernet. When we first open the Videoarchiver, we will be presented with a camera list consisting of only one camera, as shown above. We enter the IP address of one of our cameras in the IP entry box. When we ship a camera, its IP address will be 10.0.0.X, where X is the last three digits of the serial number, with leading zeros dropped. You can always reset the IP address of a camera to a universal default value of 10.0.0.34 by with the help of the camera's configuration switch. Assuming we know the IP address of a camera, we can change that IP address with the Videoarchiver's IP button. We can check communication with a camera at any time, and identify which of several we are communicating with, by turning on its white LEDs. We select "15" from the White LED menu button and the LEDs on a camera will turn on, or we will see a timeout error in the Videoarchiver text window.
The View and Record buttons cause the Videoarchiver to configure a camera for streaming or recording. It is when we start streaming video from the camera that the Videoarchiver applies our choice of resolution, rotation, color saturation and exposure compensation. We can adjust these parameters at other times, but they have no effect. We cannot change the rotation of a live display once the display has begun. We must stop the display and start it gain. We can, however, turn on and off the white and infrared LEDs at any time. The Record_All and Stop_All buttons allow us to begin and end recording from all cameras in the list. We add cameras to the list with the Add_Camera button, and we can save and load camera lists with Save_List and Load_List. The Configure button opens a configuration panel. The Directory button allows us to choose a master directory for video recording, in which the Videoarchiver will create directories named after the cameras and store its video files.
Each camera has its own View button. If the camera is state is "Idle", presswing View will create a live display of the camera view. After a few seconds, a window with the display will open. The state of the camera will switch from "Idle" to "Live". Terminate the live display by closing the window, pressing "escape" int the window, use pressing camera's Stop button. You should notice a delay of no more than one frame period between movements in the field of view of the camera and their appearance on the display. Use the live display to check that your camera is the the right place, has the correct image rotation, is focused well, and that its video configuration provides adequate resolution and contrast. We can open live displays for every idle camera in our list, by pressing all their View buttons. If we have multiple displays open, we can close them all with the Stop_All button.
Each camera has its own version parameter, by which we can select high or low resolution video. The low-reolution video will provide a higher frame rate than the high-resolution video. The following table gives the currently-defined camera versions.
We specify the quality of the compression with the "constant rate factor" or "crf", used by the H264 compression algorithm to control the extent to which moving objects will be blurred in the compressed video. At crf=1, there is no blurring at all, and at crf=51, all movement and edges in the picture are blurry. The standard value for high-quality compression is crf=23, and this is the value we use for our higher-resolution video. When we choose lower-resolution, we increase the compression quality still further with crf=15 so that we can see every pixel in the video output, and we can take advantage of the higher frame rate the lower-resolution vicdeo provides.
When we display video with the View button, the Videoarchiver scales the display by a factor display_zoom. By default, display_zoom is 1.0, which draws each video pixel on one computer screen pixel. Full-resolution images will take up most of your computer screen. Set display_zoom to 0.5 for a compact display of full-resolution video, or 2.0 for an enlarged display of low-resolution video. Each camera has its own rotation parameter, by which we rotate the image. We can set the camera down on any of its three sides without a cable coming out, and then choose the rotation value so that the image as soon as it comes out of the image sensor. The rotation parameter can be 0°, 90°, 180°, or 270°. Rotations 90° and 270° are not available on all cameras. If your camera cannot provide the rotation you request, the Videoarchiver will issue a warning.
Each camera has its own color saturation (Sat) and exposure compensation (Exp) settings. These control how the camera exaggerates color and exaggerates brightness. Higher values of provide more exaggeration. Set color saturation to zero and the video is black and white. Set it to one and the video is excessively colorful. Set exposure compensation to zer an only the brightest objects will be visible. Set it to one and the dimmest objects will be visible, while the brighter ones are washed out. The default value of both is 0.5.
Each camera has its own Record button that starts recording video from the camera. If you have a live display open for the camera, the live display will be terminated. There is also a Record_All button that will start recording from all cameras in the list. The Videoarchiver creates a directory named after the camera in the master directory. We specify the master directory with the Directory button. Because the camera names will be used as directory names, do not use spaces or special characters other than dashes and underscores in your camera names. We recommend you name each camera after its serial number, because this will simplify both your IP address assigments and your association between cameras and recording archives.
If we unplug a camera during recording, the Videoarchiver will attempt to re-start recording restart_wait_s seconds later. Any time communication with a camera takes longer than connection_timeout_s seconds, the Videoarchiver will assume that the camera needs to be re-started. When recording is interrupted, we say it is stalled, and the Videoarchiver marks the state of the camera as "Stalled". The default restart wait is 30 s, and the connection timeout is 5 s. We can view and adjust these parameter, and other Videoarchiver parameters, in the Configuration Panel, which we open with the Configure button.
The Videoarchiver keeps a log of all the times a camera stalls, and all the times it is able to re-start recording, as well as all errors and warnings issued by the Videoarchiver. We can view this log with the View_Error_Log, and clear the log with the Clear_Error_Log. The Save Configuration button saves the Videoarchiver parameters, but does not save the camera list. The Unsave Configuration button is useful for removing a configuration you don't want any more. Re-open the Videoarchiver after pressing the un-save button, and you will be back to the default settings.
The Q button provided for each camera queries the camera for its log files, and prints them out to the text window. We can use these log files to track down problems with compression and communication. The R button allows us to reboot the camera without disconnecting its power. The U button causes the Videoarchiver to update the camera's firmware. If you see errors that are not timeout errors when you communicate with a camera, it could be that the camera firmware needs to be updated to be compatible with your version of the Videoarchiver. In that case, press the U button. The updated takes less than ten seconds and does not require a re-boot.
We construct a new camera list with the Add_Camera button. We can delete cameras from a list with the X button at the end of each list entry. We can save and load lists to and from disk. The lists are saved as text files containing a sequence of Tcl commands that configure the Videoarchiver's internal camera list. When we open the Videoarchiver, it does not load a camera list automatically. We must load the list with the Load_List button, having previously saved it with the Save_List button. You can save any number of different camera lists, but if you want to load a particular camera list when you open the Videoarchiver, make sure cam_list_file in the Configuration Panel points to your camera list, and press the Save to save the Videoarchiver configuration.
We turn on and off the camera's built-in white and infrared illumination with the Wht and IR buttons. We can set them to any of sixteen power levels zero to fifteen, where zero is off and fifteen is full-power. When we use these buttons, the lamps respond immediately. If we want to adjust the lamps gradually, we use the Scheduler, which allows us to define a regular schedule of lamp adjustment, and to specify how slowly the lamps should be adjusted. When we save our cameras list to disk, our schedules are saved along with the list.
The files recorded to disk are H264-compressed videos stored within mp4-containers. The files are saved in a subdirectory of the master directory. You specify the master directory yourself, but the Videoarchiver creates the subdirectories itself. If the directories exist, it re-uses them. There is one subdirectory for each camera, and they are named the same as the camera name in the Videoarchiver camera list.
The video files are named in the format Vx.mp4, where the x is a ten-digit Unix time. The first frame of the video was generated by the camera at the start of this Unix time, as reported by the clock on the data acquisition computer. Video file V1525694641.mp4 has Unix time 1525694641, which is "Mon May 07 08:04:01 EDT 2018". The video contains twenty frames per second. Its first frame was generated some time between zero and one twentieth of a second after the start of Unix second 1525694641. Because the videos are time-stamped in this way, and the frames within the video are arriving at a constant rate, we can navigate to a particular time within a video by counting frames. The time defined by the video file is what we call video time.
We also have computer time, which is the time on the data acquisition computer clock. The computer time may or may not be kept accurate by an network time server. Whether the computer clock is accurate or not, it is what we use to set the time on the camera clocks, and it is the camera clock that dictates the segmentation of video at one-second boundaries and assigns the Unix time of the segment name. The Videoarchiver attempts to keep all the camera clocks synchronized with the computer clock to within ±25 ms. It does this by correcting the camera clock by a few tens of milliseconds every time it starts a new recoding file. With record_length_s set to 600, this correction will take place every ten minutes, and each video file has length six hundred video-time seconds.
Meanwhile, we might be recording biometric signals with SCTs and an Octal Data Receiver (ODR). The ODR has its own temperature-compensated clock that keeps time to ±1 ppm. The Neurorecorder records biometric signals in NDF file that are time-stamped in the same way as the video files. The default Neurorecorder output file is in the format Mx.ndf, where x is the Unix second that began immediately before the first sample recorded in the NDF file. Because the NDF recordings contain clock messages from the ODR, we can navigate to a particular time within an NDF by counting clock messages. The time defined by the video file is what we call signal time.
The computers used to record video and telemetry from laboratory animals are not usually connected to the internet at large, and so do not have access to a network time server. A computer clock left to run without network correction will, in our experience, drift by ±1 s/day, or 10 ppm. The ODR clock is ten times more accurate than most computer clocks. We might think to correct the computer clock using the ODR clock. But correcting a computer clock is a delicate business. When we move the clock backwards, some actions that have already taken place will be presumed not to have taken place, and will be repeated. When we move the clock forwards, some actions that have not taken place will be presumed to have taken place, and will be skipped. When a computer adjusts its time to match a network, it does so by slowly speeding up or slowing down. Even if we were willing to embark upon the development of such a synchronization algorithm for the data acquisition computer, what would we do when we wanted the computer synchronized with respect to absolute time with the help of a network server? And in any case, even a 1 ppm drift is 100 ms per day, and after a year, this amounts to half a minute. Even the ODR clock will be wrong by a significant amount eventually. If our objective is to synchronize our recordings to an absolute universal time, we must have access to a network time server. But that is not our objective.
Our objective is to synchronize signal time with video time so that we can view signals and video simultaneously in the Neuroplayer, and be sure that the two are simultaneous to within ±50 ms. It does not matter to us that the computer time is wrong, so long as video time and signal time are synchronized with one another. Our strategy is to synchronize both video time and signal time with computer time. As we have already mentioned, we synchronize the signal time on the camera clocks every ten minutes by default. The Neurorecorder, meanwhile, by default records one-hour NDF files, as measured by counting ODR clock messages. When the Neurorecorder has received exactly one signal-time hour of data, it stops recording and waits until the start of the next second of the computer clock. When the new second begins, the Neurorecorder immediately resets the ODR clock and begins recording another signal-time hour of data, naming the new NDF file after the new second. If the computer clock is a fraction of a second faster than the ODR clock after one hour, this process results in a minimal loss of data. But if the computer clock is a fraction of a second slower, we will lose up to one second of data every hour, and consecutive NDF file timestamps will differ by 3601 s instead of 3600 s.
During the recording process, the ACC is recording its own live, uncompressed video stream in one-second time-stamped segment files. It is also running several parallel compression processes that read in the un-compressed segments and produce compressed one-second segements. The first frame of each compressed segment is a key frame containing the full image. The Videoarchiver queries each camera frequently. As soon as a new compressed segment becomes available, the Videoarchiver downloads the segment to the data acquisition computer and deletes the segment from the camera. Every transfer_period_s, the Videoarchiver adds the new segments to its existing video file, until the file is record_length_s long, at which point the Videoarchiver starts a new video file. The video files created by the Videoarchiver contain a key frame every second, so we can navigate quickly and accurately to any one-second boundary within the video.
Example: The video file V1525694641.mp4 is exactly one minute long. It was constructed out of 1-s segments. It contains a key frame at the start of every second of video time. The time on the phone clock at the beginning is 08:04:01 and at the end is 08:05:01. The file itself is 7.4 MBytes, or 120 kBytes/s.
When we are viewing video and signals in the Videoarchiver, we must use a playback interval length that is a multiple of one second, or else the video time navigation will be inaccurate and slow. In order to get to a half-second video time, the video player would have to find a key frame preceeding the half-second time, and play the video until the half-second time to reconstruct the image at the intermediate moment, then stop. While it is doing this, it either displays the unwanted video, or it displays a flat color. Neither behavior makes for easy viewing, so the Neuroplayer prohibits video playback with playback intervals shorter than one second.
To investigate the effect of resolution and frame rate upon recording file size, we set up an ACC to record video of one or two people moving about on a couch, in an attempt to simulate a close-up view of animals in a cage. The following table give measured values network bandwidth required to live stream un-compressed video from the camera, and the size of the video when compressed with our standard compression quality.
|H264 Compressed Size|
|410 X 308 at 20 fps||5||1.1|
|410 X 308 at 40 fps||11||1.1|
|820 X 616 at 20 fps||15||3.1|
|820 X 616 at 40 fps||17||3.8|
|1640 X 1232 at 10 fps||16||23|
|1640 X 1232 at 20 fps||18||26|
The compressed video size does not increase in proportion to frame rate. The H264 compression carries information from one frame to the next, so that each new frame is a list of modifications to the previous frame, rather than a new presentation of an entire frame. When we navigate within a video, we move first to a key frame, and then play forward. The ACC video files have key frames at every one-secod boundary of video time, and by synchronization, these one-second boundaries are simultaneous with Neurorecorder signal time to ±50 ms
The record_length_s parameter sets the length of the recorded video files. When we add new segmets to the end of a video file, the process is not as simple as adding new data to an NDF file. We cannot simply write the new file on to the end of the old file. Instead, we must read the existing file into memory, followed by the new segments, and concatinate them to create a new, self-consistent video file. We use this to replace the original file. The longer the video file becomes, the more effort we must spend to extend it with new segments. On the other hand, the shorter the video files, the larger the number of files we have to deal with in our file system. We recommend that you leave the recording file length at 600 s, which is the default. This length limits the number of video files to six per hour, but at the same time avoids copying longer files. The length of time between segment transfers is also important. The longer we wait to transfer segments, the fewer times per hour we have to copy the video file.
We recommend transfer_period_s of 60 s. The transfer process takes roughly one second with a five-minute long video file. If we are recording from ten cameras, the transfer time amounts to ten seconds. If we attempt to transfer every ten seconds, we will have no time for downloading segments. By transferring every sixty seconds, the transfer time is kept to less than 20% of the time available.
The camera adjusts its exposure time automatically to suit the ambient illumination. We control how the camera chooses its exposure time, so as to favor bright or dark parts of the image with its exposure compensation value, shown as Exp in the controls for each camera. We can set this value between 0.0 and 1.0 with a drop-down menu to make the image darker and brighter respectively.
The exposure compensation value will be applied only when you start live capture or start recording. At other times, it will do nothing.
In the Videoarchiver, set the Sat value to something between 0.0 and 1.0 to make the image more or less colorful. A value of 0.0 produces a black and white picture. A value of 1.0 produces an image that is, in our opinion, too colorful. The default value is 0.5, which is, in our opinion, just right. Adjust saturation before you start video streaming.
The ACC comes with a lens that has no infrared blocking filter. It can see in the light of its own infrared illumination, or its own white illumination. In a room with fluorescent and white LED lighting, the camera colors will be accurate, as in the photographs above. A mixture of infrared and white light disrupts the color fidelity of the image, as shown below.
By using only overhead LED lighting during the day, and only infrared light during the night, we obtain true color images during the day and gray-scale color images at night, without any need to adjust color saturation or exposure compensation as the lighting changes.
[23-MAR-22] The Scheduler allows us to adjust the lamps on all cameras simultaneously according to a daily, weekly, or monthly schedule. Open the Scheduler Panel with the Scheduler button. When we save a camera list to disk, we also save our schedules, so they will be available the next time we open the Videoarchiver.
The Scheduler uses the same schedule-definition syntax as the Linux cron utility. A schedule is defined by five fields: minute, hour, day of month, month, and day of week. A "*" for any of these fields means "ignore this field". Five stars means "never execute the task". A numerical entry in any field defines a time when the task should be executed. Days of the week are numbered 0-6 for Sunday to Saturday, but the Videoarchiver allows you to select the days by name in its menu button, then displays the number after you make your selection.
The Scheduler operates on all cameras at the same time. When it turns on the white lamps, it turns on all the white lamps. Whenever it adjusts the intensity of a lamp, it does so one intensity step at a time. To adjust from intensity zero to intensity fifteen requires fifteen steps. The Scheduler spends step seconds at each intensity level before moving to the next. If we set step to 10 s, the Scheduler will take 150 s to increase intensity from 0 to 15.
The Verbose checkbutton turns on additional reporting in the Videoarchiver's text window. We get to see the temperature of the microprocessor on the camera, the time taken to download each segment, and the time by which the incoming segments lag behind the local clock. This lag is an indication of how well compression is proceeding on the camera. If the camera is overheating, or activity in its field of view is excessive, it is possible that the camera will not be able to compress the video stream in real time. The lag should be between four and five seconds. If it is greater than ten seconds, or is increasing steadily, something is slowing down either the compression on the camera or the transfer of the compressed segments to the data acquisition computer. The Q button downloads and prints a set of log files from the camera. These files can offer clues as to what might be going wrong when video display or recording fails. The U button updates the firmware on the camera to match your version of the Videoarchiver.
Recording Stalled: The camera state is "Stalled" with a red background. Communication with the camera has been interrupted, or there is some internal error that is preventing video streaming. The Videoarchiver will reboot the camera and keep trying to connect until it succeeds. Best thing to do is wait for a few minutes to see if the Videoarchiver can fix the problem for you. If the camera remains stalled, stop recording and try other trouble-shooting steps.
Recording Produces No Video File: Use Q to look for errors. If you see an error message declaring that a file could not be found, stop recording and use U to uupdate the firmware on the camera. Try recording once again.
Compression Lagging: Use Q to check the ACC's processor temperature. If it's 60°C or greater, the processer will be slowing its clock to avoid over-heating, video compression will low down, and the lag between recording and transfer will keep increasing. The ACC is designed to operate between −10°C and +30°C while sitting on a flat surface, or −10°C and +35°C while held in the air by a camera mount.
Compression Lagging: If the microprocessor temperature is below 60°C, check the camera view. Did you leave the lense cap on? If so, the blurred images that result present the compression algorithms with serious difficulty because there are no uniform patches of color to compress easily. Remove the lens cap. Otherwise, is there excessive activity in the field of view? Perhaps a fan moving around, or many lights flashing? Reduce the activity in the field of view. The cameras can compress images of moving animals, but not images in which everything is moving.
Internal Camera Connection Failure: When you press View or Record you subsequently see a "failed to connect to camera" or "no camera found" error returned from the ACC. This is not a failure of communication between the Videoarchiver and the ACC. It is a failure of communication within the ACC. This problem occurs roughly 10% of the time you reboot a camera that uses the libcamera libraries, such as the A3034C2. It occurs less than 1% of the time when you reboot a camera that uses the older raspivid libraries, such as the A3034B1 and A3034C1. Try the View button a couple more times to see if the camera can configure itself without rebooting. If not, reboot with the R button. You may have to reboot more than once. Once the camera is running, it will continue to run.
Timeout Waiting for Socket Connection: When you press Q, the camera does not respond. Do you see an amber activity light flashing on the ACC's Ethernet socket? If not, check your connection to your PoE switch. Unplug and plug the cable. Do the white lights flash three times within thirty seconds? Is the amber light flashing? If not, you may have damaged the ACC's Ethernet interface by plugging a LWDAQ cable into its Ethernet socket. You will have to ship the camera back to us for repair.
Timeout Waiting for Socket Connection: When you press Q, the camera does not respond, but you have an amber link light flashing on the ACC's Ethernet socket. Check the IP address you are using with the camera. We ship with IP address 10.0.0.x, where x is the last two or three digits of the ACC serial number. If you continue to have socket connection failure, reboot while pressing the brown configuration button on the camera and try address 10.0.0.34. If you still cannot connect, check the network settings on your computer: you must configure your wired Ethernet connection to run on subject 10.0.0, so that your computer will use its wired Ethernet to communcate with IP address 10.0.0.34.
Here we list changes in recent versions that will be most noticeable to the user. You will find the source code LWDAQ/Tools.