There are many things that could cause obedience or loyalty levels to go down each day, but very few related to not interacting with them. Most of these things will leave notes in the end of day reports (either Job or Secondary).
Servants with the Clingy trait generally require you to interact with them once a week or let them sleep with you in order to avoid a significant penalty to obedience and minor penalty to loyalty. Though this penalty is overall rather weak due to that lengthy delay and a 50% chance of occurring. There are many events that can occur if you do not interact with your servants for a little more than a week, but these are explicitly interactive with a variety of outcomes.
It's often easy to overlook problems with servants simply because you assume that you are treating them well. Servants with at least moderate levels of stress will occasionally lose obedience, and having high levels will occasionally cause them to lose loyalty. Having your servants work in towns where you have a bad reputation will occasionally cause them to lose obedience and loyalty.
Servants with high lust but no sexual outlet will lose a lot of obedience. Having a dirty mansion can cause all of your servants to have a chance to lose a significant amount of obedience.
Servants with high grade (Rich or Noble) are much more likely to loose obedience or loyalty and much harder to regain the obedience. Higher grades require much more luxury to keep them happy otherwise they will lose obedience and loyalty proportional to the amount of luxury they lack. Higher grades allow for higher courage and confidence, and with lower loyalty can cause obedience to drop.
