FileSystem How-To

Cross posted from our work blog.

An example of the images we were trying to save to the device... ohh dear!

Introduction

We're presently developing a mobile application using Cordova and the whole process has been brilliant! Most recently we've been using the FileSystem of the mobile device we're using (more often than not a virtual Android device using Genymotion) so we've had to get our heads around the File plugin.

My team is comprised of fullstack (with a leaning towards the LAMP and MEAN stacks) developers and front-end developers so this is why we've had some problems getting our heads around this exciting world of the FileSystem. We're used to interacting with servers and putting stuff up there and not having to think about storing things on a user's machine... apart from session cookies (and even then we'll use the language's built in abilities more often than not). To an extent it almost felt a little dirty to get into the business of storing files on a machine.

In order to get started we looked at the plugin documentation and, when that started to get a little dry, we looked at HTML5 Rocks, and that was much more exciting! It must be noted though that the FileSystem API is only available on Chrome at present, but that's cool as we primarily develop in Chrome (I often spend more time coding in the console rather than my IDE).

While it was exciting we came across any number of head-scratching moments so I've decided to document them here, I hope it helps!

Getting access

It's all well and good to be able to play with the FileSystem but how do you get started? The first thing to do is to ask whether or not you can play!

Because we're developing on a virtual device and the Chrome browser (thankfully it's the main browser I use for development as - as I noted above - it's the only major browser to support the FileSystem API) we need to ask permission nicely. This did cause us some problems to start with but the thing to remember is that Chrome doesn't support the proper requestFileSystem function but has its own. It's not a major issue but it's still a concern. Another thing to bear in mind is that your Cordova app should always include the cordova.js library (it won't be there unless the app is built but if it's the last external JavaScript file you call then it's not an issue in development), because the cordova.js file adds a cordova object to the window object we can test on that to tell whether or not we need to do the substitution for Chrome. This is in the main file of our project and it works a treat:

var requestedBytes = 1024*1024*10; // 10MB
if(!window.cordova){
    window.requestFileSystem = window.requestFileSystem || window.webkitRequestFileSystem;
    navigator.webkitPersistentStorage.requestQuota (
        requestedBytes,
        function(grantedBytes) {
            window.requestFileSystem(
                PERSISTENT,
                requestedBytes,
                function(fs){
                    window.gFileSystem = fs;
                    fs.root.getDirectory(
                        "base",
                        {
                            "create": true
                        },
                        function(dir){
                         console.log("dir created", dir);
                        },
                        function(err){
                         console.error(err);
                        }
                    );
                },
                function(err) {
                    console.error(err);
                }
            );
        },
        function(err) {
            console.error(err);
        }
    );
}else{
    document.addEventListener(
        "deviceready",
        function() {
            window.requestFileSystem(
                PERSISTENT,
                requestedBytes,
                function(fs) {
                    window.gFileSystem = fs;
                    fs.root.getDirectory(
                        "base",
                        {
                            "create": true
                        },
                        function(dir){
                         // alert("dir created");
                        },
                        function(err){
                         alert(JSON.stringify(err));
                        }
                    );
                },
                function(err){
                    alert("Error: ", JSON.stringify(err));
                }
            );
        },
        false
    );
}

This is somewhat lengthy but I like the indentation as it gives me a nice indication of what's happening and where. In the function which uses the Cordova plugin we're using alerts whereas we use console.logging on Chrome. In order for the alerts to make sense I'm converting the err objects into strings. You can do this yourself now to test by simply entering this in the console (just so long as you're reading this in Chrome):

window.requestFileSystem = window.requestFileSystem || window.webkitRequestFileSystem;
navigator.webkitPersistentStorage.requestQuota (
    1024*1024*10,
    function(grantedBytes) {
        window.requestFileSystem(
            PERSISTENT,
            1024*1024*10,
            function(fs){
                window.gFileSystem = fs;
                fs.root.getDirectory(
                    "base",
                    {
                        "create": true
                    },
                    function(dir){
                     console.log("dir created", dir);
                    },
                    function(err){
                     console.error(err);
                    }
                );
            },
            function(err) {
                console.error(err);
            }
        );
    },
    function(err) {
        console.error(err);
    }
);

Hopefully you'll see an alert asking for permission to use Local data storage and, once you've granted access, the console should print out these 2 lines:

undefined
dir created

The undefined means that the function doesn't return anything and it isn't anything to worry about but the dir created means we've created a base directory How cool!

In order to better see the effects of what you've done download the HTML5 FileSystem Explorer Extended Chrome (FileSystem Explorer) plugin. Once you've installed it you'll be able to see an empty base directory in your permanent storage. If you have a way of serving files (and who doesn't?) you could do worse than looking at the HTML5 Rocks page I linked to above and running the sample code. It's really rather cool!

Working with Directories

I like to keep things somewhat organised, I guess I'm something of a neat freak, but that's okay really as it means I don't often lose things. As such I'd like to keep the files I want to work with organised. We also want our application to be enclosed in one folder so that if/when we make another app we can distinguish between documents hierarchies. So we'll create a base folder as soon as we get told we can play with the FileSystem. Within that folder we can create other folders. This way we can create a whole taxonomy of our applications concerns and we'll be able to CRUD things later confident that we know where things should go.

This process is somewhat philosophical though so I'd suggest giving it some thought. Coming from a LAMP background means that I'm all in favour of normalisation and believe in repeating myself as little as possible, being thrust into the world of MEAN means that I'm coming to embrace the "messy" approach of replicating data as and when needed as well as expanding database rows as I need to. This does require something of a shift from rigid to flexible thinking but getting my head around both ways of working has helped me become less programmatically autistic.

Once we've got a reference to our base directory we can start creating a folder tree with base being the trunk. This then allows us to create document leaves. One thing to remember though is that we can get directories and files… and we can get a reference to them even if they don't exist by telling the API to create them if they don't exist. But we do need to be careful about creating documents in folders that don't yet exist and this is the one big gotcha I'd like to point out to you in this piece. I'd like to tell you how we got around it too.

First though let's look at an example of getting a file and creating it if necessary:

gFileSystem.root.getFile(
    "base/test.txt",
    {
        create:true
    },
    function(file){
        console.log("Created File", file);
    },
    function(err){
        console.error(err);
    }
);

Again, you'll have to excuse the formatting (this is the last time I apologise about it though - I know we could do this as a one-liner but I like this passing of functions and I really do think functions deserve their own line… and if functions deserve their own lines surely objects and strings do too?). Hopefully if you enter this into the console you'll get a nice message saying Created File and an FileEntry object. Go ahead and expand that FileEntry object and check that it has these attributes: filesystem: DOMFileSystem, fullPath: "/base/test.txt", isDirectory: false, isFile: true and name: "test.txt".

That's grand isn't it? We've created a file... but we might want to write to it. We can do that by using a FileWriter on the FileEntry:

gFileSystem.root.getFile(
    "base/test.txt",
    {
        create:true
    },
    function(file){
        console.log("Got File", file);
        file.createWriter(
            function(fileWriter) {
                fileWriter.onwriteend = function(progress) {
                    console.log("Write completed", progress);
                };
                fileWriter.onerror = function(err) {
                    console.error("Write failed", err);
                };
                var blob = new Blob(
                    ['Lorem Ipsum'],                     {
                        type: 'text/plain'
                    }
                );
                fileWriter.write(blob);
            },
            function(err){
                console.error("Error creating writer", err)
            }         );
    },
    function(err){
        console.error(err)
    }
);

If you now navigate to base/test.txt in FileSystem Explorer and click on the file Chrome should open an new file with "Lorem Ipsum" in it. Cool ehh?

There are all sorts of other things that you can do with a fileWriter like append lines or save binary data like images. Have an explore but do remember that once you have a reference to the file the world's your lobster!

Gotcha

So we can write to a file once we have a reference to it, whether or not it exists, and we can use directories once we have a reference to them, whether or not they exist… what about getting a reference to a file that doesn't yet exist within a directory that doesn't yet exist. This is the major issue we dealt with and had us stumped for quite a while. If you replace base/test.txt with base/test/test.txt you'll get a lovely FileError object on the console with these attributes: code: 1, message: "A requested file or directory could not be found at the time an operation was processed." and name: "NotFoundError".

Clear as mud ehh?

Basically we need to make sure that the directory we're looking in exists before we can look for the file because while we set create to true for the getting of the file, we don't set create to true for the directory because we aren't using getDirectory. Phew! HTML5 Rocks has a lovely function for just this situation called createDir which accepts a DOMFileSystem object and an array representing a directory path (i.e. in our example: ["base", "test"]):

var path = 'base/test';

function createDir(rootDirEntry, folders) {
    // Throw out './' or '/' and move on to prevent something like '/foo/.//bar'.
    if (folders[0] == '.' || folders[0] == '') {
        folders = folders.slice(1);
    }
    rootDirEntry.getDirectory(
        folders[0],         {
            create: true
        },         function(dirEntry) {
            // Recursively add the new subfolder (if we still have another to create).
            if (folders.length) {
                createDir(dirEntry, folders.slice(1));
            }
        },         function(err){
            console.error(err);
        }
    );
};

createDir(gFileSystem.root, path.split('/'));

This is brilliant but we needed a reference to a file so we could write to it so we came up with this:

var path_file = "base/test/test.txt";

function createDirAndFile(rootDirEntry, folders, callback) {
    if (folders[0] == '.' || folders[0] == '') {
        folders = folders.slice(1);
    }
    rootDirEntry.getDirectory(
        folders[0],         {
            create: true
        },         function(dirEntry) {
            if (folders.length > 2) {
                createDirAndFile(dirEntry, folders.slice(1), callback);
            }else{
                callback();
            }
        },         function(err){
            console.error(err);
        }
    );
};

createDirAndFile(
    gFileSystem.root,     path_file.split('/'),
    function(){
        gFileSystem.root.getFile(
            path_file,
            {
                create:true
            },
            function(file){
                console.log("Got File", file);
                file.createWriter(
                    function(fileWriter) {
                        fileWriter.onwriteend = function(progress) {
                            console.log("Write completed", progress);
                        };
                        fileWriter.onerror = function(err) {
                            console.error("Write failed", err);
                        };
                        var blob = new Blob(
                            ['Lorem Ipsum'],                             {
                                type: 'text/plain'
                            }
                        );
                        fileWriter.write(blob);
                    },
                    function(err){
                        console.error("Error creating writer", err)
                    }                 );
            },
            function(err){
                console.error(err)
            }
        )        }
);

It's lovely because we use recursion to create the directory structure (in the same way as createDir did) before writing the file… it wasn't of a great deal of use in our most recent application as the hierarchy is really quite shallow but I think it will be of use later on. Go ahead and try it in the console, you should end up with the expected directory structure when you look at it with FileSystem Explorer.

Conclusion

The FileSystem API ROCKS! It can cause something of an existential crisis for those of us who not used to working with a FileSystem - except on the server - but once you get your head around it it really does make a lot of sense. Most of the articles use an error handler to deal with the errors that can occur but, if you're looking at the console trying to debug what went wrong where, the error handler function generally doesn't give you a line number. This is really, really annoying and we've ended up commenting out the internals of our functions to try and track down the bug, so we went back to adding the console.logs as methods to the relevant functions.