The Silent People are homeowners who don’t speak up at meetings (and generally don’t attend, giving their representation via written proxies), send in emails, or express their opinions with their names on them. Buck said that these folks are afraid of retaliation if they express themselves directly, so they use live-person proxies like Irve, Walker, and Buck to represent them secretly. The unstated implication is that they are people who have much more at stake financially than the rest of us, so they can’t afford to stick their necks out.