I've been playing with WebSQL of late (after all, why wouldn't you play with a deprecated standard?). And I'm loving it for inserting data via Papa Parse, the thing is, it was proving to be really, really slow... right up until I clocked that I was doing it wrong!
The thing is, I was iterating over my data and firing a transaction with an SQL payload for each row of the CSV file I was parsing where I should've been firing a transaction and then iterating over my data and firing off multiple SQL payloads within the single transaction.
This was my first effort:
for (let record of results.data) { mydb.transaction( t => { t.executeSql(` INSERT INTO accounts ( Date, Company, Account, Amount ) VALUES ( ?, ?, ?, ? )`, [ record.Date, record.Company, record.Account, record.Amount ], t => { let query = "SELECT COUNT(*) AS total FROM accounts"; t.executeSql( query, null, (t, result) => { if (~~result.rows[0].total === tableMetaData.total) { createTable(); } }, null ); }, null ); } ); }
Pretty, eh? But it took silly amounts of time to execute and I was planning on inserting 250,000 records. It simply wasn't practical to do it that way as it took something like 20 seconds to insert 1000 rows:
Rows | Time |
---|---|
10 | 0.218 |
100 | 1.711 |
1000 | 21.111 |
5000 | 90.703 |
Changing the code to this:
mydb.transaction(t => { for (let record of results.data) { t.executeSql(` INSERT INTO accounts ( Date, Company, Account, Amount ) VALUES ( ?, ?, ?, ? )`, [ record.Date, record.Company, record.Account, record.Amount ], null, null ); } }, error => { console.log("error", error); }, () => { createTable(); } );Produced these results:
Rows | Time |
---|---|
10 | 0.081 |
100 | 0.058 |
1000 | 0.083 |
5000 | 0.223 |
Not only quicker but I've also halved the number of queries as I don't need to check that all the transactions have completed after each insert, sweet eh? After figuring this out I came across a number of posts where the same thing had been discovered by others. Including RICAL and Gavin Pickin. Just a shame it's deprecated.
No comments:
Post a Comment