When a user has access to a group that provides only View Items permission and not Edit Items permission, the Edit button for those users is still enabled when viewing items even though the fields are disabled for Editing. The Edit button should either also be disabled or removed. Also ensure that the user permission to edit is validated on submission and not only at the UI layer.
It seems that the Edit button remains active, to allow the user to assign the item to one of their folders. Since the folder assignment interface is only exposed on the Edit page, the user needs to be able to access the Edit page.
So, to implement your suggestion, a significant change to the UI would need to be made.