GZipping JSON for localStorage

By Suparerg Suksai

I've written before about trimming JSON before but I've recently been thinking about how we can trim it even further.

I came across the fantastic pako earlier in the week and decided to try it out with some data. You'll see from the repo that we're testing two different JSON files, one was quite small (and was, indeed, the example I used earlier), the other is significantly larger.

If you download and run the repo, you'll see I've added a check to see if the data retrieved from localStorage is the same as the data retrieved from the file system - it is!

You'll also notice something interesting in the developer console: the smaller JSON ends up being larger (119.93%) when gzipped compared to the stringified JSON; the larger JSON file was 37.49% the size of the stringified JSON.

That sort of makes sense, though, doesn't it? Gzipping a file adds some overhead (hash table), and that overhead might end up making the final gzipped file larger. Not that I know a great deal about the process: I'm cribbing this from an answer on Quora.

The compressAndStore process is quite interesting, with these steps:

1. The JSON is fetched form a .json file as a string
2. The fetched string is ocnverted into a JSON object
3. The JSON is stringified (IKR - we've only just converted into an object)
4. The stringified JSON is defalted with pako
5. The resulting Uint8Array is converted into a regular array
6. The array is saved to localStorage, along with the original stringified JSON

async function compressAndStore(file, name) {
  // Get our data
  const response = await fetch(file)
  const fetchedJSON = await response.json()
  // Convert our JSON to a string
  const stingifiedJSON = JSON.stringify(fetchedJSON)
  // Deflate our data with Pako
  const deflatedStringifiedJSON = pako.deflate(stingifiedJSON)
  // Convert the resulting Uint8Array into a regular array
  const regularArray = Array.from(deflatedStringifiedJSON)
  // Store our data (both deflated and the original)
  localStorage.setItem(`${name}Array`, JSON.stringify(regularArray))
  localStorage.setItem(`${name}JSON`, stingifiedJSON)
}

The retrieveAndDecompress process is almost a direct reverse:

1. The array is retrieved from localStorage as a string
2. That string is converted back into an array
3. The array is converted into a Uint8Array array
4. The Uint8Array is inflated with pako
5. The inflated Uint8Array is decoded and then converted back into a JSON object.
6. The original file is compared with the retrieved and decompressed file

async function retrieveAndDecompress(file, name) {
  // Get our data for later testing
  const response = await fetch(file)
  const fetchedJSON = await response.json()
  // Get our data from localStorage
  const retrievedData = localStorage.getItem(`${name}Array`)
  // Convert it into an array again using JSON.parse()
  const retrievedArray = JSON.parse(retrievedData);
  // Convert the array back into a Uint8Array array
  const retrievedTypedArray = new Uint8Array(retrievedArray);
  // inflate the Uint8Array array using Pako
  const deflatedTypedArray = pako.inflate(retrievedTypedArray)
  // convert it back into the original data
  const json = JSON.parse(new TextDecoder().decode(deflatedTypedArray))
  console.info(`Is the fetched ${file} the same as the retrieve and decompressed ${name}Array: ${JSON.stringify(fetchedJSON) === JSON.stringify(json)}`)
  const regularArraySize = (localStorage[`${name}Array`].length * 2) / 1024
  const stingifiedJSONSize = (localStorage[`${name}JSON`].length * 2) / 1024
  console.log(`${name}Array (${regularArraySize}) is ${((regularArraySize / stingifiedJSONSize) * 100).toFixed(2)}% of the size of ${name}JSON ${stingifiedJSONSize}`)
}