I don't know if the attention would affect the image quality, but did you compare your Iterations per second vs XFormersUsing the merge I made and messing about with the sampler and scheduler and some prompts I managed to get slightly better images.