In this case I would simply say: j'ai aimé passer du temps avec toi. In my understanding, J'ai bien aimé... is a weaker kind of liking. Something along the lines "it was ok passing time with you".

I thought “aimer” means love, so I wanted to “weaken” it to “aimé bien.” So to clarify my question, does “aimer” mean love only when it is directed at a person (as opposed to a thing or activity where it would mean “like”)?

It means “love” also when directed at an object. If you want to express something weaker, I’d go with your second option.