I went upstairs in the bandit house and a prompt told me "We should clear out the bottom floor first" before letting me up anyway
Edit: Went down and cleared the remaining 3 bandits out and went up again. The encounter against all three repeated
Edit: Went down and cleared the remaining 3 bandits out and went up again. The encounter against all three repeated