Andrew hesitates, lifting his head from James's shoulder to look at the other Andrew. Andrew returns his gaze with the same veiled look of trepidation. Neither replies for a few moments.

"You said the Master didn't want to go back to being one person," starts one.

The other finishes for him. "...What was it like? I mean, he's still in there, isn't he? You remember being him?"

They're both Andrew, but wouldn't going back together mean that they'd sort of...stop existing? Or that one of them would? It's irrational (more obviously so from an outside perspective), but they're afraid of being subsumed by one another.