go downhill; be on the decline; 走下坡路
At that time our relations were going upwards, but since then they have been going downwards [downhill]. 那时我们的关系是走上坡路, 此后就开始走下坡路了。