In each of my business relationships, I expect that mutual respect and trust be essential ingredients in my association with the other individual. If one or both traits do not exist, then the relationship shouldn't proceed any further.
So, what do you do when you have an uncomfortable or odd feeling about working for someone, but you can't put your finger on it? Should you continue the business relationship or move on?
I really cannot answer these questions for you, but I have learned that in my many years of working for or with people that it is simply fine just to move on. In other words, if I believe that a business relationship is not mutually satisfying, than it is okay to end it. There are plenty of employers out there and plenty of other projects to work on. The same can be said about the other person: if you leave them or they drop you, they will find someone else.
In my opinion, you need not have a specific or tangible reason either. Sometimes you have a gut reaction to a particular project while other times there may be something about the project that simply goes against your principles or just doesn't sit well with you. No matter, simply end the business relationship and move on.
How you end the relationship is up to you. If you want to leave a door open, telling the person that you are busy with other projects is fine. If you want to shut the door, you can tell them specifically why you no longer want to work for this person.
In all cases, season your words with kindness, but don't waffle and certainly don't tell lies. You can't worry about what others think about you; to do so is a waste of time and will certainly impact your ability to develop new and solid business relationships down the line.