How do I deal with version codes for APKs in Android?

Yes, you need to publish before or while building.
However, whether or not the existing APKs will download the new JavaScript depends on things like the SDK version number etc. This flowchart might help.

If this is not what you want there are a few things you can do.

Look into release channels:

https://docs.expo.io/versions/latest/distribution/release-channels/
https://docs.expo.io/versions/latest/distribution/advanced-release-channels/#promoting-a-release-to-a-new-channel

Also, using the Updates module you can disable automated updates and manually check for updates yourself. You should be able to use the returned manifest to decide whether to download the update or not.

See also the following posts on versioning:

This one and this one are more relevant to ejected apps, but the description of how he uses release channels might still be useful.