At one extreme end of the spectrum, some might say we can do whatever we want with our property. For example, if I want to start a chicken farm in my residential neighbourhood or right next to a patio restaurant, that's my business because it's my property. And if the chicken farm doesn't make enough money, maybe I could convert it to a commercial helicopter landing pad or a coal power plant, a casino, or even a rock-n'-roll stadium. I'm sure the neighbours would appreciate me.
Others would argue that the right to private property is conditional on its impact beyond its boundaries.
Another argument could apply to social advantage. For instance, let's suppose I live in an expensive neighbourhood and my neighbour decides to convert his property into a large apartment complex to alleviate the shortage of affordable housing in the community. On the one hand, this project could hurt my property value. On the other hand, to prohibit him from building the complex could prevent housing from becoming more affordable for the local poor, or alternatively force them to live farther away from work and so have to spend more time and money on transportation. In such a case, when city bylaws interfere with the free market to the detriment of the poor, what obligation does it then have to compensate the poor for such interference in the market?
My personal view would be that the government has a right to regulate the first set of examples above as that has to do with quality of life for local residents. However, I'd tend to oppose government interference in the second example above as it involves the government interfering in the free market to the explicit benefit of the rich over the poor, so as to artificially maintain high property values.
Of course there could be many other viewpoints, but where would you draw the line to the right to private property?