You are thinking of push and pops to stack, as memory goes via stack no endian conversion happens.
So I think this might be a bit implementation specific, since different runtimes represent stack items differently. E.g. py-evm:
vs geth:
But this is getting into the weeds a bit on specific implementations which can vary between clients.
Yes, definitely a little in the weeds + implementation specific. But all clients for LE architectures must do conversion sometimes (whether they do it “eagerly” a read/write from memory or “lazily” when something like ADD is requested, unless there are some really funky bigint implementations that I’m not aware of 
), so I think it’s worth mentioning. Even on a BE arch, you still need to copy 32 bytes, because memory is mutable. The very best you can do is create a stack item which is copy-on-write.
Hence the suggestion that it is included in the memory expansion cost at a x2 rate:
e.g. memory use = memory slots + 2 x tstore slots
Rather than having its own load factor
It’s a thought, although I was under the impression that the quadratic pricing for memory is evil 
and we wanted to move away from it, see for instance Proposals to adjust memory gas costs
Since the superlinear pricing in both memory and transient storage is intended to prevent DOS, I think another way to think about it is: how should the practical memory bound increase as gas limit increases? I think DDR and L1-L3 cache do not scale over time a la Moore’s law the same way as CPU speed and storage cost (see references). I haven’t done the analysis, but I suspect it is something like log() or sqrt() compared to the above two variables, and this EIP bounds memory like sqrt(gas limit). We could try something to bound like log(gas limit), but I think that makes the pricing model more complicated in terms of math.
References:
Historical Memory Prices 1957+
Historical cost of computer memory and storage - Our World in Data