I’ve drafted an EIP which has linear pricing, and but avoids the subcall upper bound invariant: EIP: Linearize Memory Costing
Per @qizhou’s comments, it might make sense to add a (transaction-)global limit to the memory usage, on the order of 64-128MB.