bags question>mounting

ok, i have this whole thing apart ready to flip the ball joints and bolt on my bag brackets, got a few questions: does the sway bar stay or go?? I pulled the clip and castle nut out, but the ball joints arent budging?!? Words of wisdom?