You could use this expression but the tense is wrong it should be-
Ils m'avaient permis de les accompagner = They had allowed me to accompany them
and it is more formal than -
They had let me come with them = Ils m'avaient laissé venir avec eux
Hope this helps!