You will, for a while. If you pay off all your loans, and close your credit cards your score will go to zero (well technically "undeterminable") over time. Last figures I saw is it takes about a year for that to happen.
Also, paying off your debt can cause you score to drop just from paying it off. I saw this when we paid off our house.
Here's an article that explains.
Basically your FICO score is determined from these factors, some impacting it for good, some for bad:
•35% of your score is based on your debt history.
•30% is based on your debt level.
•15% is based on the length of time you’ve been in debt.
•10% is based on new debt.
•10% is based on type of debt.
Here's an article that explains it pretty well.
Now, personally I would be fine being "undeterminable" as I no longer borrow money (haven't for years) but have not let mine drop just because it is pretty easy to maintain it after we paid off our house as I mentioned in my earlier post.
I should add I am not a financial advisor, just a random dude on the Internet explaining how it has worked for me. It may not work for you.
Now I have to stop talking about this before @
Tcoat has to pull out MooseGirl on me!