How long does it take to lose fitness?
What do you mean by "losing fitness"? That's too vague.
If you mean getting fatter, then if depends entirely on your calories in and calories out. If you eat in a calorie maintenance (eating as many calories as you burn) You'll never get fatter. Be careful, the amount of calories you burn depends on a lot of things such as your amount of physical activity, your metabolism and your age, which affects your metabolism. (the older you are, the slower your metabolism is)
If you mean losing muscle, as far as I know, you can stop doing any physical activity and you can still hold muscle for up to 3 weeks. If you want to preserve muscle and not build more, you can just decrease your effort in the gym to a third of what you kneed to grow muscle. BTW your muscles have muscle memory so even if you do lose muscle, you can regain it much easier.
If you are talking about your cardiovascular health, you can stop doing all physical activity for a week without losing any progress.