I have noticed that with the NPC factions, you see a red message that if you open a container or harvest a crop it will decrease your standing. Now when you accidentally try to open a container you get a warning that says something like: Doing this will lower your faction, do you really want to do this? I think that harvesting on a plot should have the same warning, unless you choose to continue. That would be very helpful at the beginning as all the foremen and farmers are usually standing in between the crops. If you attempt to harvest a crop and select not to the warning should pop up every time for that faction. If you choose to loot the crop the warning goes away.
No joke but in these scenarios I have resorted to lining my mouse up on the trader/npc and taking my hand off my mouse before I will press F. It is that bad. Some of the plantations and farm POIs are pretty bad about this.