It's been a few years since I read the books, but as far as District 12 and its rampant poverty and food shortages, I never necessarily got the strictest impression that it was the only impoverished district. But it was all that Katniss, our narrator, knew. It's worth remembering that Katniss is, to varying degrees, an unreliable narrator throughout the books, either due to outside factors (isolation and ignorance of other districts) or due to her mental state.
Also, with 12 being furthest from the Capitol, "out of sight, out of mind" is likely a big factor. And (IIRC) it is one of the s