You ask, What makes it worth defending? and the only answer I can give is this: Freedom to write, freedom to read, freedom to own material that you believe is worth defending means you're going to have to stand up for stuff you don't believe is worth defending, even stuff you find actively distasteful, because laws are big blunt instruments that do not differentiate between what you like and what you don't, because prosecutors are humans and bear grudges and fight for re-election, because one person's obscenity is another person's art.The entire essay is in response to a reader asking why we should defend lolicon manga, and I agree 100% with everything he says. I've seen lolicon in some of my searches (I'll admit a guilty pleasure in seeking out erotic-yet-not-vile doujins, and I find I have to wade through 5-10 distastful and icky ones involving lolicon, incest, or rape to find one that I don't find objectionable) and yes, it's quite gross. I think the world would be a better place if it didn't exist, if only slightly. Even still, I would not for a moment consider banning it or making it illegal, and would actively vote against any politician who did.
Because if you don't stand up for the stuff you don't like, when they come for the stuff youdo like, you've already lost.
I can't make the argument any better than Mr. Gaiman did. Go read his article, it's worth it.