In fiction, does it matter if the facts are wrong?

A few years ago, I tried to read a popular, well-reviewed novel by a North Carolina author who shall remain anonymous. Not a quarter of the way through, I found myself plowing through a scene set at the house of the protagonist's grandmother.
I'm figuring it was August or early September. School was in session, but it was hot. Tomatoes plants were bearing. The main character had set out to mow Grandma's lawn, and she cautioned him to look out for her just-planted peony and tulip bulbs.

Whoa. I stopped there and read the passage again. I'm no expert gardener, but I know enough to know:
A. Peonies aren't bulbs.
B. You probably wouldn't plant tulips until fall.
C. If you had planted bulbs, they'd be underground, so you couldn't mow them down.
As I kept going, I found more errors involving wisteria and pinks. I never finished the book.

I thought of that novel recently as I listened to Charlotte author Andrew Hartley speaking to a local book club about his novel, "The Mask of Atreus." As he began talking about it, he confessed that it contained an error. The book is set in Atlanta. Hartley knows the city well -- he lived there for years. But his main character drove to Athens, Ga., on the wrong highway.
A minor error, yes. But the fact that he brought it up told me he still felt bad about it. As a reporter, I know that the occasional error is inevitable, even when you're really careful. Hartley told me later that he researches like crazy. But the mistakes usually come when you're writing about something you know about.

"What’s maddening about these things," he told me in an e-mail, " is that they are completely inconsequential for the story, often affecting no more than the phrase in which they appear, but they can knock people out of the story, so they are a real concern."

That's exactly what happened to me when I discovered the gardening errors. Suddenly, I was thinking about bulb planting times instead of plot. Chatting with Hartley gave me new respect for fiction that transports me to another world. There's a ton of work that goes into making make-believe believable.

But even the best writers make errors. Hartley shared a fascinating one: In William Golding's "Lord of the Flies," the character of Piggy wears glasses because he's nearsighted. The allegorical novel centers on a group of British schoolboys stranded on a deserted island. The boys use Piggy's glasses as a magnifying lens to start fires. Here's the problem: The kind of lens used to correct nearsightedness doesn't focus sunlight and wouldn't start a fire.

Do mistakes in fiction drive you crazy? Ever encountered errors in well-known novels? Let me know.