Outlier/jump detector functions?

[mihaispetrescu]

Hi,

Great project! I was wondering if there's any interest in adding robust statistics methods to remove outliers or detect frequency jumps.

[aewallin]

Yes, there is already one gap-robust adev function, but more are welcome. Preferrably based on some published papers or established algorithms. Also think about how to test new functions i.e. what datasets to use and how to test that the algorithm is actually working. Then also think about how new functions fit to the API - are they filters that run on the data before calling a standard adev() function or how should it work?

[mihaispetrescu]

There are some algorithms in stable32 that would be useful, and can be ported. I think stable32 uses median absolute deviation for detecting outliers. For detecting jumps, I've used sequential and block averaging, and Riley has a pretty good write-up on pros/cons. It's quite possible they may already be implemented in python/numpy somewhere, but they wouldn't be hard to write up if not. An algorithm like DAVAR can be useful (it may be implemented in your package already, i apologize if i haven't seen it!).

[mihaispetrescu]

Also, I've seen Dr. Tavella papers floating around that work with other techniques.

I'd love to contribute to your project, and look forward to your thought.

[aewallin]

Dynamic adev, median absolute deviation, and jump detection would all be good additions to the library. It's good if there is a paper as a reference (describing the algorithm and maybe giving a sample implementation), some test dataset (real or synthetic) and some reference results to test against (e.g. Stable32, if what it does is documented well enough).
If you have time to work on allantools you could also consider finishing the bias-correction work and/or the confidence interval work that is now half-finished until I or someone else has time and energy to finish it.