In the Generation I games, Generation II games, and Pokémon Ruby, Sapphire, and Emerald, the amount of money lost is simply half of the money the player has. In Pokémon FireRed, LeafGreen, and all games from Generation IV onwards, the money lost is calculated using the following formula:
Money lost = Level x Base payout
Where Level is the level of the player's highest-level Pokémon and Base payout is looked up using the following table:
(Table doesn't lend itself to pasting very well but tldr it's based on # of badges)
The minimum amount of money that the player can lose is $8 (if the Pokémon with the highest level in the player's party is level 1 and the player has 0 Badges; 1×8=8), while the maximum is $12000 (if the Pokémon with the highest level in the player's party is level 100 and the player has 8 Badges; 100×120=12000).