It mentions on day 3 that Rayford wants to see you and "best not to keep him waiting, he's a bastard that gets easily annoyed."
Being as I'm an Asshole in this quest, I decided to wait a full day and see him. He mentions the "fight with Gorth yesterday" although that was two days ago. I was hoping to score some Infamy points by annoying him, or get a death sequence by having him kill me for disrespecting him. Ah well.
A simple solution would be to auto-set Roehm meeting up with Rayford, or add in a reaction from him if you keep him waiting.
(Playing through the second time I'm experimenting with stuff I didn't do the first time through.)