pith. sign in

arxiv: 1812.09372 · v1 · pith:FN34QY5Fnew · submitted 2018-12-21 · 💻 cs.DS

Fast post-hoc method for updating moments of large datasets

classification 💻 cs.DS
keywords momentslargedatadatasetdatasetsrequiresupdatingmean
0
0 comments X
read the original abstract

Moments of large datasets utilise the mean of the dataset; consequently, updating the dataset traditionally requires one to update the mean, which then requires one to recalculate the moment. This means that metrics such as the standard deviation, $R^2$ correlation, and other statistics have to be `refreshed' for dataset updates, requiring large data storage and taking long times to process. Here, a method is shown for updating moments that only requires the previous moments (which are computationally cheaper to store), and the new data to be appended. This leads to a dramatic decrease in data storage requirements, and significant computational speed-up for large datasets or low-order moments (n $\lesssim$ 10).

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.