Each of the four iPhone sizes will make the same layout feel quite different.

I generally do my initial sketches/wires at the SE size because it tells me the maximum amount of UI I can comfortably accommodate.

In the visual phase, any time I arrive at a solution that shows promise, I do a quick "sanity check" across multiple screen sizes, rather than flushing out an entire design for one size.

I often find that when spacing and scale of elements looks good at one screen size it will feel two small or large at other sizes. Sometimes I have to settle on a scale that doesn't look optimal for any one screen size so that it looks acceptable on all screens.