Looking for advice on persisting data quickly on app exit. Specifically using both Expo.SecureStore and Expo.FileSystem.writeAsStringAsync I need to persist state and large JSON objects before suspension or app quit. Is anyone else doing something like this and how do you handle it?
What I am doing now: I call AppState.addEventListener('change', ...) in my top level component and watch for ‘background’ state change. Then I trigger my saves. I load it back up when I get an ‘active’ or when the app starts.
Issue: I am getting data loss. I assume this is a case of data corruption or some such. These saves work fine when the user does not exit the app and I can trigger the saves while the app is still running. But the Expo.SecureStore are lost and my files (stringify-ed JSON) can not be found. Well, to be accurate this only happens randomly on some devices but I think it’s a timing issue. The faster apps are finishing the save before the app fully exits (pure assumption here).
Use case & more info: For those wondering why. We are a data capture SaaS company. Our clients enter about 14,000 data points in our app per day. These are simple integers that are stored in JSON format that we stringify and persist in files. The users are offline and exclusively on cheap android devices (slow tech). At the end of the day they connect and upload the files to our servers. We proved the APK outside of the expo app itself but make no changes and even use the expo build:android tool. The issue is, users leave the app in the middle of an incomplete form and lose data.
Personally I want to tell them to deal and use the app as designed but it is an interesting problem and I was hoping someone had an idea of how to improve this.
I think your intuition that it’s getting forced closed before it can finish the write is correct. You best bet is to see how to essentially guarantee that no additional data is being saved on app state change. Once your app goes inactive, there’s no real guarantee how long it will be in memory. Hopefully it has time to finish all its tasks, but, you really never know.
A few ideas:
- Save automatically every X seconds, or after X seconds of inactivity (in addition to how you save already). You could measure inactivity by hooking into, say, TextInput usage.
- Chunk into smaller files. Is it a single file per day? If so, those are big files that could take a while to save.
- SecureStorage isn’t really meant for huge JSON blobs, so I’d avoid it for that. Maybe AsyncStorage is better, but, at the end of the day, depending on your device, it may just be saving your data to a…
- SQLite database! I think this may be your best bet in the long run. Every time a user adds a data point, insert it into a local database. Export that to a file when you need it in a file. If by “connecting”, they’re plugging in a USB cable and pulling the file from the disk, you’d need some button in the UI to generate the file, I suppose. You could also have the same button upload the file to a server if your infrastructure allows it.
Expo has a SQLite API that looks and works like basically every other SQLite API put out in the last decade or so. Such API’s generally interact with a SQLite driver tightly integrated into the OS. SQLite is designed to deal with atomic writes efficiently and safely. Your users shouldn’t see a performance issue or a data loss issue if you’re writing every input to a database as it’s entered.
1 Like
Thanks @llamaluv! Sorry for the late response. Its good to get some confirmation on my suspicions.
Your first recommendation of using automatic saving over time is a great one. A bit of a facepalm moment for me. We do chunk it into a few smaller files but ultimately they are still quite large. Uploading in smaller segments might not be a bad idea as well, though we have some technical reasons that would not be ideal. The SecureStorage is really only being used to store groups of vars so we are good there.
We did actually use SQLite about 4 years ago (a few app generations back). The issue is we found the SQLite databases actually had a very small total data cap. Our total data usually exceeded the limits. I am not sure this is still the case and I am not sure if the speed issue has been resolved (SQL writes would lock our UI).
In essence thanks for the response and your first idea helps. It also good to see that the way we are doing it is not so very far from what others might do in the same situation (excluding SQLite).
Welcome!
RE: SQLite database limits, were you possibly using WebSQL in a Cordova app? In such a case I think you would have been limited to 25-50 MB. I’m not aware of any SQLite storage limit for a React Native app (which will always use the native driver) or a Cordova app that uses the native SQL driver (requires a plugin).