Determining the optimal parameter values for a scene parsing network architecture is an important task as a network with optimal parameters produces the best performance. A manual selection of parameter values requires expert domain knowledge of the intrinsic structure and extensive trial and error, which does not guarantee optimal results. An automatic search of the optimal parameters is desirable to harness the full potential of a scene parsing framework. The network architecture needs to be evaluated with various combinations for several parameters to achieve optimum performance. We propose a stacked three-level deep context-adaptive end-to-end network. The end-to-end network architecture extracts visual features, and contextual features, and optimally integrates both features in three logical levels for scene parsing. Particle Swarm Optimization (PSO) algorithm has been used to efficiently search for optimal solutions. Using PSO, we set an optimal set of parameters in all three levels to obtain optimum performance. The PSO aims to optimize the network globally by considering all hyperparameters in the network to achieve the best performance.