That just sounds like you don't have DJI Go4 set as your default.
Here are Instructions from Litchi to clear the defaults.
After you do that you could reset Go 4 as the default app to open whenever you connect the controller.
If you have another app (such as DJI Fly/Go) set as the default for the remote controller USB connection, you will need to clear the defaults for that app in your mobile device (Settings - Apps - DJI Fly/Go (or other app name) - Clear Defaults). Some devices automatically set the app you start as default, so this step may have to be done every time you want to switch app (for example from DJI Fly/Go to Litchi and vice versa)