jeudi 28 mai 2015

New to python: My method to import CSV to SQlite DB

I just started python programming and find it very useful so far coming from a Delphi/Lazarus background.

I recently downloaded trend data from a SCADA system and needed to import the data into a sqlite db. I thought I would share my python script here.

This process would have taken a lot more programming in Pascal. Now I just create a GUI with Lazarus and use TProcess to run the script with some parameters and the data is in the db.

Sample of trend data

Time,P1_VC70004PID_DRCV,P1_VC70004PID_DRPV,P1_VC70004PID_DRSP
6:00:30,27.75,3000,3000
6:01:00,27.75,3000,3000
6:01:30,27.75,3000,3000
6:02:00,27.75,3000,3000
6:02:30,27.75,3000,3000
6:03:00,27.75,3000,3000
6:03:30,27.75,3000,3000
6:04:00,27.75,3000,3000
6:04:30,27.75,3000,3000
6:05:00,27.75,3000,3000

Python code:

import csv
import sqlite3
import sys

FileName = sys.argv[1]
TableName = "data"
db = "trenddata.db3"

conn = sqlite3.connect(db)
conn.text_factory = str  # allows utf-8 data to be stored

c = conn.cursor()
c.execute("DROP TABLE IF EXISTS " + TableName)
c.execute("VACUUM")

i = 0
f = open(FileName, 'rt')
try:
    reader = csv.reader(f)
    for row in reader:
        if i == 0:
            ## Create Table header section from Header info in CSV doc
            c.execute("CREATE TABLE %s (%s)" % (TableName, ", ".join(row)))
        else:
            ## Import row data into database
            c.execute("INSERT INTO %s VALUES ('%s')" % (TableName, "', '".join(row)))

        i += 1

    conn.commit()

finally:
    f.close()
    conn.close()
    print("Imported %s records" % (i))

Aucun commentaire:

Enregistrer un commentaire