dimanche 17 janvier 2016

Replacing SQL data with trend line points?

I have SQL data table in which I have two columns, which I can plot as X and Y to see their relationship. This data is generated continuously, so unless I do something to reduce it I will have issues of maintaining and searching huge table.

So what I would like to do is to periodically remove all data in my data table and replace it with some finite number of points which would represent a trend line generated from the previous data. And eventually, as more and more data is generated, and after I am performing same operation, the final data will be corrected and Y would move up or down in some points related to X.

There is example of my column X and column Y plotted in XY plot, as well as an example of my desired result after running this data reduction: Blue: original data, Red: Reduced data

I have tried to do something like this before:

SELECT avg(Y), X FROM logData GROUP BY ROUND(X,2)
SELECT avg(Y), X FROM logData GROUP BY X HAVING X BETWEEN 0.9*X AND 1.1*X

However, it didn't really work. I am more looking for a way to find average Y for array of X sections.. like 0.000 to 0.019, 0.020 to 0.59, ....... 2.00 to 2.20 etc. So instead of having 10,000 points, I might have just 500 or 1000 well distributed along X.

I am using SQLite, however, I am really look for any type of SQL solution, since I don't know how I would approach this. Does anyone know how this can be done?

Do I just need to do it 'manually'...e.g., by getting min and maximum values of X, generating my array of ranging where to average Y and repeat so many queries writing to the new table, which will be used at the end to replace data in original table?

Or can this be done in a single query statement?

Aucun commentaire:

Enregistrer un commentaire