iOS builds with old build numbers crash on restart after minor version bump

Some preliminaries:

  • Managed workflow
  • eas-cli/0.41.1 darwin-x64 node-v16.3.0
  • expo: ~43.0.3
  • expo-updates: ~0.10.13 (possibly relevant)

I’m still very new to Expo, so not exactly sure what I should include here.

We deploy using EAS and are currently only focused on iOS. We’ve run into an issue where all 1.0.* builds crash on the second startup; i.e. I can install any 1.0.* build using TestFlight, and it runs fine; completely functional. As soon as I close the app and restart, it crashes and will never open again. However, 1.1.* builds work fine, even though there is very little difference between the first of the 1.1.* builds and the last of the 1.0.* builds.

As a test, I pulled down the commit from the 1.0.56 build. I renamed the build and version number to 1.1.4 and pushed it up to TestFlight. This version works fine - no crashes after closing the app.

Notably, the 1.1.* builds do include a new package in package.json, react-native-svg. If that were the problem, I wouldn’t expect simply renaming 1.0.56 to 1.1.4 to have resolved the issue.

To reiterate - literally the only difference between 1.1.4 and 1.0.56 is the expo.version number and expo.ios.buildNumber in app.config.ts. Yet one functions completely fine and the other crashes.

This is puzzling.

My understanding is that EAS doesn’t publish JS updates automatically, and we currently don’t use expo publish. I bring this up because I found something interesting in the expo-updates documentation:

If your app.json does not contain an updates.fallbackToCacheTimeout field it will default to 0 and expo-updates will attempt to start your app immediately with a cached bundle while downloading a newer one in the background for future use. When using this configuration, users that download the app from a store and launch it for the first time will always see the version of the app that the binary was built against.

We do have updates.fallbackToCacheTimeout set to 0, but like I mentioned, we don’t use expo-updates, at least not intentionally. However, the above could explain what’s going on. The version of the app that the binary was built against seems to always work on install and first use - it’s perhaps only after an update is attempted in the background and the user closes and reopens the app that the app continually crashes.

Any leads or ideas here would be much appreciated - this has been one of the most frustrating and difficult problems to debug.

Although I haven’t found it useful, I’ve included a sample crash report below.

Incident Identifier: F7A6DC05-DFB7-4626-844A-C343F355A826
Hardware Model:      iPhone12,1
Process:             LocalKitchens [1251]
Path:                /private/var/containers/Bundle/Application/A322E09C-F475-4C3E-8027-8E0A75D0DC69/LocalKitchens.app/LocalKitchens
Identifier:          com.localkitchens.guest-mobile.ios
Version:             1.0.56 (1.0.56)
AppStoreTools:       13C90b
AppVariant:          1:iPhone12,1:15
Beta:                YES
Code Type:           ARM-64 (Native)
Role:                Foreground
Parent Process:      launchd [1]
Coalition:           com.localkitchens.guest-mobile.ios [819]

Date/Time:           2022-01-26 14:24:24.6739 -0800
Launch Time:         2022-01-26 14:24:16.2050 -0800
OS Version:          iPhone OS 15.2.1 (19C63)
Release Type:        User
Baseband Version:    3.01.02
Report Version:      104

Exception Type:  EXC_CRASH (SIGABRT)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Exception Note:  EXC_CORPSE_NOTIFY
Triggered by Thread:  33

Last Exception Backtrace:
0   CoreFoundation                	0x1808450fc __exceptionPreprocess + 220 (NSException.m:200)
1   libobjc.A.dylib               	0x19907fd64 objc_exception_throw + 60 (objc-exception.mm:565)
2   LocalKitchens                 	0x1021a5134 RCTFatal + 668 (RCTAssert.m:146)
3   LocalKitchens                 	0x1022226f8 -[RCTExceptionsManager reportFatal:stack:exceptionId:suppressRedBox:] + 600 (RCTExceptionsManager.mm:89)
4   LocalKitchens                 	0x1022230b4 -[RCTExceptionsManager reportException:] + 1532 (RCTExceptionsManager.mm:164)
5   CoreFoundation                	0x1807ce3a4 __invoking___ + 148
6   CoreFoundation                	0x1807ebb74 -[NSInvocation invoke] + 468 (NSForwarding.m:3378)
7   CoreFoundation                	0x1808229d4 -[NSInvocation invokeWithTarget:] + 80 (NSForwarding.m:3475)
8   LocalKitchens                 	0x1021d5fa0 -[RCTModuleMethod invokeWithBridge:module:arguments:] + 460 (RCTModuleMethod.mm:584)
9   LocalKitchens                 	0x1021d8438 facebook::react::invokeInner(RCTBridge*, RCTModuleData*, unsigned int, folly::dynamic const&, int, (anonymous namespace)::SchedulingContext) + 540 (RCTNativeModule.mm:181)
10  LocalKitchens                 	0x1021d8070 operator() + 56 (RCTNativeModule.mm:103)
11  LocalKitchens                 	0x1021d8070 invocation function for block in facebook::react::RCTNativeModule::invoke(unsigned int, folly::dynamic&&, int) + 100 (RCTNativeModule.mm:95)
12  libdispatch.dylib             	0x1804b5924 _dispatch_call_block_and_release + 32 (init.c:1517)
13  libdispatch.dylib             	0x1804b7670 _dispatch_client_callout + 20 (object.m:560)
14  libdispatch.dylib             	0x1804bedf4 _dispatch_lane_serial_drain + 672 (inline_internal.h:2601)
15  libdispatch.dylib             	0x1804bf968 _dispatch_lane_invoke + 392 (queue.c:3937)
16  libdispatch.dylib             	0x1804ca1b8 _dispatch_workloop_worker_thread + 656 (queue.c:6727)
17  libsystem_pthread.dylib       	0x1f121b0f4 _pthread_wqthread + 288 (pthread.c:2599)
18  libsystem_pthread.dylib       	0x1f121ae94 start_wqthread + 8

Hi @michaelparlato,

Thanks for the report and for detailing this issue – it sounds frustrating!

I’d like to figure out if there are any updates at all on your project. Can you run expo publish:history and see if there are any updates present? Also, just to check, we recently came out with EAS Update. You can see if you have any updates there with eas channel:list, eas branch:list and eas update:list. I would expect the output of all of these to have no data.

Also, locally I use a program named Proxyman (like Charles Proxy) to see network traffic. That would also show if there’s an update being downloaded when you don’t expect there to be one.

For some context, updates check for three things to determine compatibility: an SDK version (called a runtime version with EAS Update), a platform, and a release channel match (EAS Update uses “channels” and “branches” to determine if an update should be available to a build). Based on your findings, it makes me think it may not be update-related, since app.version and the app.ios.buildNumber are not used when checking for compatible updates. --But no matter what it is, id love to get to the bottom of it.

Let me know how this goes!

Thanks for the reply!

So there are updates under expo publish:history and one of them looks suspicious. See attached screenshot.

The January 25th update for 1.0.57 is especially concerning because TestFlight actually doesn’t have a 1.0.57 build available (we never released such a version - I can see a 1.0.57 on Expo builds but it is an iOS simulator build released by one of our developers for testing I think).

eas channel:list returns nothing
eas branch:list returns nothing
eas update:list isn’t a command I have, eas update:view exists, but needs a groupId.

So I’m curious what the implications are here - it could be that since the 1.1* builds are all “higher” than 1.0.57, none of them are encountering this issue? But all builds lower than 1.0.57 are?

Also wondering if deleting the 1.0.57 build from Expo could resolve the issue for all of our users on 1.0.* version.

I should also note that January 25th, the day of the 1.0.57 publication, is the day that all of our users began experiencing crashes. Seems like fairly conclusive evidence :joy:

Correction: we do have a failed iOS App Store build for 1.0.57, back on January 22. But again, this version was never actually released on TestFlight.

I think we’re onto something! I wonder if the 1.0.57 update contains calls to a package (like react-native-svg) that the 1.0.18 builds do not contain, and that’s what’s causing the crash.

I think you could fix this be rolling back the latest update:

expo publish:rollback --release-channel default --sdk-version 43.0.0

When running that, it will take the latest 1.0.18 updates and move them to be the latest updates, so that when a end-user’s app fetches an update, it’s no longer the 1.0.57 update. I’d love to hear how this goes!

PS: we are currently working on a new updates service named EAS Update that addresses issues like this, and will eventually come with some UIs to make rolling back/seeing the state of your project a lot easier. (Introduction to EAS Update - Expo Documentation) Currently, it’s in “preview” only for paying customers, but will be available to everyone (free users included) later this year.

Ok, I ran the following:

$ expo publish:rollback --release-channel default --sdk-version 43.0.0

[1]  + 20786 suspended  expo publish:rollback --release-channel default --sdk-version 43.0.0

Publish history still looks the same though. Is there an execution time for that command to complete?

For me, that command showed a lot of output:

Are you only seeing the following output?:

[1]  + 20786 suspended  expo publish:rollback --release-channel default --sdk-version 43.0.0

Also might be worth making sure that expo whoami outputs the account you expect, just to make sure it’s not a permission issue. (unlikely but worth a shot)

If rolling back continues to not work, a work around would be to check out a commit before the 1.0.18 commit, then run expo publish again, effectively reverting (re-publishing) the update before the “bad” 1.0.57 update

Hm, I ran expo whoami, saw expected output, and ran the rollback command again. It worked.

Also tested one of the failed builds - 1.0.56 - it works!

Thanks for the help @jonsamp :raised_hands:

Anytime. If you have more questions or feedback about Expo (especially updates, which I work on), please feel free to email me (jon@expo.dev).

By the way, this output looks like what might happen if you immediately press Ctrl-Z after running the command and before it can actually do anything. :man_shrugging:

Posting an update for posterity’s sake :slight_smile:

An important note is that, in the happy path, we use EAS to build and publish.

Recap

  • We had an accidental publication of our bundle using the Expo updates feature when a developer ran expo build without fully understanding the implications
  • All of the builds that were published prior to that update (2022-01-25) ended up downloading the update on first use. When the user opened the app for the second time, it would crash

To resolve this, I attempted to rollback the published update, using the expo publish:rollback command.

This had an unintentional effect:

  • The rollback meant that the new “latest update” was an old build from (2021-12-17), prior to most of the app’s functionality being implemented. So, after the rollback, all versions (rather than just those published prior to 2022-01-25) would download the old, faulty build. Essentially the rollback broke the app for all users.

The fact that our app is built using EAS so that we can use features like Stripe Apple Pay, Sentry, and FullStory, meant that there was no way to push a new, working build with expo publish (at least this is my understanding). And we could not use EAS updates because the released apps were not instrumented for that.

The resolution was to delete all published updates so that at least new installs would not lead to crashes (this required manual effort from the Expo team - huge shout out for the timely response).

A few take aways:

  • Most importantly, use a production release channel rather than default so that pushing updates requires more explicit action on the developer’s part
  • Set expo.updates.enabled: false in app.config.js to avoid accidental updates if the feature is not being used

Some feedback for Expo:

  • The documentation on updates should note that updates are downloaded based on publish date. Probably the biggest surprise to me here was that after rolling back the published update, when I downloaded build 1.1.5 from the App Store, the app downloaded an “update” for build 1.0.18 which was several months outdated and had only 20% of the app’s full functionality. My expectation would be that an app can never consume an “update” which was published prior to the version the app was built against. I understand there is some complexity here related to rollbacks.
  • I did not know it was possible for a developer that doesn’t even have access to our Apple Developer account to push updates to our production apps. It seems to me there should be more guardrails here.
  • We were flying blind as to what was happening. More visibility into published updates in the Expo developer UI would have been useful. For instance, the crash reports we were seeing don’t mention build 1.0.57, but that was the version in the update. Some indication in our Expo account that apps were attempting to download that version would have helped immensely
1 Like

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.