According to Down East Magazine's article "Where's Cabot Cove?" in the August 1999 issue, "Jessica Fletcher's hometown sure looked a lot like Mendocino, California ... even if it was meant to be Boothbay Harbor." Well, Down East should know these things!
In the series Cabot Cove's exact location on the Maine coast was never fixed, and it tended to shift around according to the writers' whims, kind of like a modern day Brigadoon. Hints are sprinkled throughout the episodes set in Maine - a mention of how long it takes to drive to Portland, for instance, or the mileage between it and New York City. One episode even briefly shows a sign post grouping Cabot Cove with the names of several other real towns in Maine.
It should be remembered, though, that these details were generated by people in living in California, not Maine, and as such they can't reasonably be used as a map. Besides, hardly any of these hints agree with each other anyway - a little continuity flaw that is best overlooked.
Another hint can be found in Martin Gottfried's authorized biography of Angela Lansbury, Balancing Act. There in Chapter 18 it is stated that the producers based Cabot Cove on the real town of Castine, Maine. This is all well and good, but I've been to Castine, home of the Maine Maritime Academy, and this town, which has a very strong identity of its own, doens't much resemble the Cabot Cove of Murder, She Wrote.
Thus, we turn to the wisdom of Down East and its claim that Cabot Cove is Boothbay Harbor, a town of similar size and character. I think Down East has got it right, but decide for yourself: below are images of both Mendocino, California (Cabot Cove's stand-in) and Boothbay Harbor, Maine.
Boothbay Harbor, Maine
Mendocino, California