You just say no, journalists aren’t the police, they don’t have any special powers, they’re just regular people working for a news organisation. It’s important to query your reasons for declining.
Don’t jump to the conclusion that it’s because you have done something wrong (unless you have), the vast majority of interview requests are from journalists looking for insight, help and expertise, not combat.