I'm actually surprised I haven't written specifically about flashbacks, but I did complain about it for a bit in this post.
One or two flashbacks doesn't bother me enough to warrant complaining about them, but it get ridiculous when the flashback takes up more than one episode. Kimi no Iru Machi has certainly calmed down on all the past events it would show, but this anime has been one of the few that has gone overboard with its flashbacks.
And I understand why they do it -- mainly to setup backstory for an event about to occur -- but the first part of flashback is the word flash, which we assume to be quick. However, the term itself has been mangled quite thoroughly as time went on, with different iteration made by different stories and anime.
So, what makes these types of backstory building scenes so annoying? Well, the best way to explain it is to compare it to time skips; which is essentially the same thing, only the flow of time is reversed.
Generation Gap
When we are watching a story progress, it is to be expected that we are moving forward in time. So when we are introduced to something like a time skip, it seems almost like a more natural way of introducing the character, become invested, and then see them older -- plus the skip is permanent, so you wont end up with weird flow of time issues.
With flashbacks, I always feel like I'm getting nowhere, simple because you halt the main plot line to get dragged into something from the past -- even if the purpose is to introduce some backstory. And this is where the distinction between Kimi no Iru Machi the manga and it's anime begin to split.
Where most of the anime's flashbacks happened, the manga had these things established before they go to the Tokyo Arc. Now, I know that anime and manga are going to be different, but having a flashback WITHIN a flashback is where anyone would draw the line; and the manga itself had the flashback, but it was only one level deep, compared to the anime taking it to the absurd two levels deep.
Getting off Kimi no Iru Machi, there was one anime that pulled off a mix of time skips inside a single flashback, that being Shinsekai Yori.
Some people would argue that the novel was better than the anime, but I haven't read the original, so I wouldn't know. Either way, I'm pretty sure the structure is similar, where Saki is talking about the events that happened in her younger years; which eventually get to the present day of when she was telling said story.
So even if the entire story itself is technically a big flashback, the plot is moving forward all the way through.
Going Forward
It would be easy to say never use flashbacks, but there is a tasteful way to use them. And since they are useful, and several stories do use them, they shouldn't be completely eliminated. I just wish there were more stories that didn't heavily rely on long flashbacks to establish a characters backstory. Maybe start the characters out young and let the viewer piece together their progress would be a better option.