In order to facilitate the comparison of this gauge data to a NOAA primary station, I wanted to automate the task of determining the higher highs and lower lows. A six-minute averaging scheme was developed in Matlab by my friend Val in order to generate a six-minute record similar to the NOAA format. This still left us with squared peaks, so I decided to apply a filter to the data.
Problem: I wanted to use filtfilt, which filters data in the forward direction and then again in the reverse direction, resulting in a zero phase distortion. I tested filtfilt out on the full 12-second water level data and everything seemed hunky dory. The problem came when I tried it out on the 6-minute averaged data. The filter seemed to run okay with no errors, but the result was all NaN (no data) values. At first I thought it was caused by zeros in my vector data, so I changed all 0.00 into 0.01. This did not solve the problem. To make the problem even more frustrating, filtfilt worked on every other column of data coming out of the 6-min-averaging scheme except water level (time, std, etc.). What is up with that?
Work around: I used the convn function in Matlab in order to apply a convolution to the data. This filter can cause a phase distortion, however, by specifying the shape as 'same', the central part of the convolution is returned and the distortion is minimized.
For the comparison of our preliminary gauge to the NOAA primary gauge in Bar Harbor, a slight phase distortion will not affect the results. In order to really be correct, however, a filter that guarantees no phase distortion should have been used. If anyone has any ideas why filtfilt would not work on a running-averaged dataset, I am all ears. Even one of my profs could not figure it out.
I must say though, the results of my code using convn do look pretty nice on first inspection. Rounded peaks and automatically picked highs and lows (using the extrema function, modified so that returns are in linear time order, not descending):
Posts
3 comments:
I found your blog via googling for 'matlab filtfilt nan', since I was having the same problem. It turns out one of the entries in the unfiltered array was -inf, and changing it to 0 fixed the problem. (I can imagine 0 could be bad, too, so if that doesn't work, maybe try something else.)
I found your post in an identical way to John (previous comment).
I was filtering a dataset which contained no NaN's, no inf's, etc. Filtfilt was giving no error, yet the result was all NaN's. If I filtered a small part of the dataset (e.g. first 100 points) the filtered series 'blew up' quickly.
After an extremely frustrating couple of months doing numerous work-arounds, I finally found my problem.
My input data series was of type 'single', rather than 'double'. To change the data to double precision, just use
>> double(indataset)
This worked for me, maybe it might work for you. The 'single', 'double' property may have been set by the way the data was recorded on the tide gauge.
Phil.
This update is long overdue, but I just wanted to thank both John and Phil for their suggestions.
Here is what I have tried:
I had a few 0 values, so I tried changing them to 0.0001 instead. This did not work. I also tried removing my negative values (only 3 of them) just to see if that helped for any reason, it didn't.
The column of data I am attempting to use filtfilt on is already catergorized as a double. I tried reassigning it just in case and then reapplied filtfilt and again got all NaNs.
Right now this issue is on the back burner for me (as the project is over and the work around seemed to do the trick), but I know this is going to come back and haunt me at some point.
Post a Comment