> Is it just me or do other climbers think Secret Swinger is dangerous?
Nope, not just you.
The two most popular routes on Lower Baldy are probably No Complications and Secret Swinger. Both are a bit tough for their advertised grades because the ground below them has eroded. Each has a hard start consisting of a jump to a giant flexing choss-jug and heel-hooking wrestling match, the difficulty of which is disproportionate to the rest of the climb. One day one of these jugs will come off and someone will have an accident.
If I were going to do routes on Upper Baldy I would instead walk straight up to the half-way ledge. The quality of the rock and climbs on Lower Baldy is poor and you will not be missing much. I have personal experience of quite a lot of 'Ghoul' falling down.