Lily Potter was abused by James

Where did Harry's parents get all of their money from?

It never really explains where the money Harry's left over comes from. His grandparents on both sides are apparently dead before the first book begins, possibly from the Death Eaters.

In the Hagrid book and movie it says, "You didn't really think your parents would leave you with nothing?" Again, it's not clear what James and Lily did for a living if they ever had to work for a living (perhaps James was actually rich independently; he's cocky enough as a kid to be an heir), but if they worked, you probably would have some sort of retirement fund that they'd retired into and that would be the next of kin to Harry.

Also, while the Potterverse financial system is not extensively covered, it would be easy to imply that Gringott pays interest on any gold they are allowed to use in borrowing. Some safes are like lockers, while others are more like regular bank accounts. Gringotts is a business like anything else; I doubt the goblins love gold so much that they would work for free just to be around. Check out the wonders of compound interest that have been created over 10 years on a nest egg that will never be touched.

Anyway, what's left for Harry is enough that Harry's time at school is financially comfortable, but it's by no means extravagant; For example, he didn't buy a Firebolt of his own. It just looks like a lot compared to what the Weasleys live on; The Muggle Artifact Abuse Office obviously doesn't pay much to work in the Muggle Artifact Abuse office, and the Weasleys have seven children. The Malfoys, on the other hand, are the bad Rockefellers; so much money that Lucius doesn't believe in buying Draco's way onto the Slytherin Quidditch team.