Respected commentator Garry Lyon wrote on Wednesday that it was time for Fremantle and Pavlich to part ways for the good of all concerned as the club was as far away from challenging for a premiership as it has ever been.
Harvey said the club's critics should be a lot more measured in what they say.
"To be speaking about a guy like that in those terms, [at] round four, we're in a vulnerable situation because we haven't won a game," Harvey said from Fremantle Oval on Wednesday.
"It's really too easy for guys in really high profile positions in the media to dissect a club and pull the club apart.
"From afar it can always be easy to have a look at Fremantle and quickly pull the trigger on everything about the club."
With Fremantle winless after four rounds and coming off a 14th place finish in 2008, Harvey stressed Pavlich's importance to the club as a leader and a player while it rebuilds.
"When you look at Matthew Pavlich, he will go down as one of the greatest players Fremantle's ever had," Harvey said.
"He's captain of the football club. So for us to be talking about trading him [is] bewildering."
Harvey also addressed the growing criticism of young midfielder Rhys Palmer's disposal following his game in the 83-point loss to St Kilda.
Palmer hit the target with 12 of his 20 disposals on Saturday, and has an efficiency of 68.5 per cent for the season so far. The AFL average is 75 per cent.
Harvey said extra opposition attention and the scrutiny that follows the NAB AFL Rising Star winner were reasons for Palmer's early-season skill woes.
"We forget quite quickly that Rhys is only in year two of his career," the coach said.
"You're never the complete player until you're playing regularly and you're sort of four or five years down the track and the opposition are really on guard about what you do."
Palmer has averaged more than 21 touches a game, but Harvey says he has only been playing 60 per cent game time.
"Looking at his output, he's still having a lot of the ball in a short period of time.
"He's certainly still racking up possessions. It's just the quality that needs improving."
Harvey said draftee Matt de Boer and young ruckman Zac Clarke were both in the selection mix for this Saturday's Anzac Day clash with the Sydney Swans.
"This is going to be a huge stoppage game, so you need thick-set ruckmen," he said when asked if Clarke may debut.
"That's something we're weighing up at the moment."
