Pretty sure he was talking to Nagina, since when the go out there is a hisssAfter the reveal, I for some reason assumed he might have been talking to a guard (or janitor?) as opposed to a (imaginary) prisoner, and they were unhappy with their job or something. Didn't even consider that the joke was he was crazy and imagining things, lol.