Abstract: The relay protection equipment can quickly and reliably remove the fault parts in the power system, which largely depends on the accurate protection setting. In order to give full play to the performance maximize the capacity of relay protection equipment, combined with the conditions of selectivity and sensitivity that need to be met in actual operating conditions. The setting calculation of inverse time overcurrent protection is transformed into a nonlinear optimization problem with multiple constraints and variables. An optimization model of inverse time overcurrent protection setting is established. On this basis, in order to solve the defect that the existed particle swarm algorithm is easy to converge prematurely. A new hybrid algorithm (DEPSO) is proposed as a solution method for the optimization model. The benchmark function is used to optimization comparison test, which verified the effectiveness and reliability of the improved algorithm. The simulation examples show that improved hybrid differential evolution algorithm can effectively enhance the performance of inverse time overcurrent protection.