When asked for whom he wrote, W.H. Auden said it was for people who liked his work. I can not think of a better answer than that.
Perhaps it is a tad ingenuous but it also true. Those are our readers. We can’t worry about all of those who might not like what we do. We can’t tailor our work to woo them.
We can only try to improve, to better our craft, to hone our ability to communicate. Maybe we can even be one of those people who like our work.
But I wouldn’t count on that.