Optimizer parallelism also referred to as zero redundancy optimizer [37] implements optimizer point out partitioning, gradient partition