jeudi 9 avril 2015

Python 3.4 - How to transform a Pandas Data Panel (not frame) to a mySQL database?

I am trying to organize some financial data in a 'multidimensional' SQL database so that I can, at a later stage, take slices as needed across time or asset (or attribute such as 'close_price').


Pandas Panel.to_sql seemed like a nice way to do this (albeit a lack of detailed documentation on data panels specifically), and so I have managed to store all the data in a pandas data panel:


http://ift.tt/1JtnfIn



<class 'pandas.core.panel.Panel'>
Dimensions: 1322 (items) x 2717 (major_axis) x 15 (minor_axis)
Items axis: 0 to 1321
Major_axis axis: 2004-01-02 00:00:00 to 2014-12-24 00:00:00
Minor_axis axis: Open to name


In this case the Items are Security IDs (or stock symbols), Major_axis are the dates, and Minor_axis hold the various attributes (prices etc.)


I tried (assume 'dp' is the name of this data panel and 'path' is the target path for the database):



from sqlalchemy import create_engine
import sqlalchemy

engine = create_engine('sqlite:///'+ path)
dp.to_sql(name = 'Equities_Data', con = engine, flavor = 'sqlite')


This flagged an error that the dtypes were not recognized - and I noticed that .to_sql takes an additional argument dtypes, a dictionary which maps pythonic types to sqlalchemy types. Fine so did this:



attributes = list(dp.minor_axis)
values = [sqlalchemy.types.String etc.. ] (populated this with loop)
dtype = dict(zip(attributes, values))


This still did not solve the problem for transforming the full datapanel but did manage to allow me to transform a dataframe slice to a .db file.


i.e.: dp[0].to_sql(name = 'Equities_Data', con = engine, flavor = 'sqlite', dtype = dtype, index = dp.major_axis)


So I guess my question is: has anyone managed to get this sort of thing to work in the past? Is this even possible? Easier ways? I am all ears.


Otherwise, if I loop create all 2D slices of my panel as databases is there a way to combine these in SQL to form a 3D-like object where I can slice through as I want?


Thanks so much in advance.


Aucun commentaire:

Enregistrer un commentaire