The idea behind EfficientPS is influenced by our goal of achieving superior performance compared to prior state-of-the-art models while simultaneously being fast and computationally efficient. Initial panoptic segmentation methods heuristically combine predictions from state-of-the-art instance segmentation network and semantic segmentation network in a post-processing step. However, this disjoint approach has several drawbacks including large computational overhead, redundancy in learning and discrepancy between the predictions of each network. Although recent methods have made significant strides to address this task in top-down manner with shared components or in a bottom-up manner sequentially, these approaches still face several challenges in terms of computational efficiency, slow runtimes and subpar results compared to task-specific individual networks.

We address the aforementioned challenges in our EfficientPS architecture that provides effective solutions to these problems. EfficientPS consists of our new shared backbone with mobile inverted bottleneck units and our proposed 2-way Feature Pyramid Network (FPN), followed by task-specific instance and semantic segmentation heads with seperable convolutions, whose outputs are combined in our parameter-free panoptic fusion module. The entire network is jointly optimized in an end-to-end manner to yield the final panoptic segmentation output.

Previous panoptic segmentation architectures rely on ResNets or ResNeXts with Feature Pyramid Network (FPN) as the backbone, which consume a significant amount of parameters and have a limited representational capacity. In order to achieve a better efficiency, we propose a new backbone network consisting of a modified EfficientNet that employs compound scaling to uniformly scale all the dimensions of the network, coupled with our novel 2-way FPN. We identify that the standard FPN has its limitations to aggregate multi-scale features due to the unidirectional flow of information. Therefore, we introduce the novel 2-way FPN that facilities bidirectional flow of information which substantially improves the panoptic quality of foreground classes while remaining comparable in runtime.

We incorporate our proposed semantic head with separable convolutions that captures fine features and long-range context efficiently as well as correlates them before fusion for better object boundary refinement. We build upon Mask R-CNN for the instance head and augment it with separable convolutions and iABN sync layers. One of the critical challenges in panoptic segmentation deals with resolving the conflict of overlapping predictions from the semantic and instance heads. In order to thoroughly exploit the logits from both heads, we propose a new panoptic fusion module that dynamically adapts the fusion of logits from the semantic and instance heads based on their mask confidences and congruously integrates instance-specific foreground classes with background classes to yield the final panoptic segmentation output.

We evaluate EfficientPS on four challenging urban scene understanding benchmark datasets, namely Cityscapes, Mapillary Vistas, KITTI and IDD. EfficientPS is ranked #1 for panoptic segmentation on the widely used Cityscapes benchmark leaderboard, exceeding the prior state-of-the-art model by a large margin, while consuming fewer parameters, lesser computation and faster inference time. In addition, EfficientPS is also ranked #2 on the Cityscapes semantic segmentation benchmark as well as the Cityscapes instance segmentation benchmark, among the published methods. EfficientPS consistently achieves state-of-the-art panoptic segmentation performance on Mapillary Vistas, KITTI and IDD benchmark datasets. EfficientPS is the first to benchmark on all the four standard urban scene understanding datasets for panoptic segmentation and exceed the state-of-the-art on each of them while simultaneously being the most efficient model. Achieving computationally efficient rich and coherent image segmentation has widespread implications for image recognition systems that have to make sense of cluttered real-world environments, where objects move and overlap. Segmenting foreground objects together with background is important for understanding entire scenes and for performing related actions, such as navigating through dynamic scenes.