I was trying to say 'you don't like them' (ie dogs) so I said 'tu les n'aime pas. Is the wrong?

It's almost correct, Helen. The word order is a bit different and the verb needs to be in second person singular, because of the "tu":

Tu ne les aimes pas.

-- Chris (not a native speaker).

what is the rule why the word order needs to be different?

The word order in French is different than English. For e.g I ask you will be je te demande..... And not je demande toi....