Saturday, 22 April 2017

WebSQL bulk insert

I've been playing with WebSQL of late (after all, why wouldn't you play with a deprecated standard?). And I'm loving it for inserting data via Papa Parse, the thing is, it was proving to be really, really slow... right up until I clocked that I was doing it wrong!

The thing is, I was iterating over my data and firing a transaction with an SQL payload for each row of the CSV file I was parsing where I should've been firing a transaction and then iterating over my data and firing off multiple SQL payloads within the single transaction.

This was my first effort:

for (let record of results.data) {
    mydb.transaction(
        t => {
            t.executeSql(`
                INSERT INTO
                  accounts (
                    Date,
                    Company,
                    Account,
                    Amount
                  )
                VALUES (
                  ?,
                  ?,
                  ?,
                  ?
                )`,
                [
                    record.Date,
                    record.Company,
                    record.Account,
                    record.Amount
                ],
                t => {
                    let query = "SELECT COUNT(*) AS total FROM accounts";
                    t.executeSql(
                        query,
                        null,
                        (t, result) => {
                            if (~~result.rows[0].total === tableMetaData.total) {
                                createTable();
                            }
                        },
                        null
                    );
                },
                null
            );
        }
    );
}

Pretty, eh? But it took silly amounts of time to execute and I was planning on inserting 250,000 records. It simply wasn't practical to do it that way as it took something like 20 seconds to insert 1000 rows:

RowsTime
100.218
1001.711
100021.111
500090.703

Changing the code to this:

mydb.transaction(t => {
    for (let record of results.data) {
            t.executeSql(`
                INSERT INTO
                  accounts (
                    Date,
                    Company,
                    Account,
                    Amount
                  )
                VALUES (
                  ?,
                  ?,
                  ?,
                  ?
                )`,
                [
                    record.Date,
                    record.Company,
                    record.Account,
                    record.Amount
                ],
                null,
                null
            );
        }
    },
    error => {
        console.log("error", error);
    },
    () => {
        createTable();
    }
);
Produced these results:
RowsTime
100.081
1000.058
10000.083
50000.223

Not only quicker but I've also halved the number of queries as I don't need to check that all the transactions have completed after each insert, sweet eh? After figuring this out I came across a number of posts where the same thing had been discovered by others. Including RICAL and Gavin Pickin. Just a shame it's deprecated.