|Generally it'll be the cobbles that pull out, and depending on their size and the height of the cliff, that could be anywhere from an annoyance to a major danger to the belayer. I'd suggest abseiling over (with a helmet and eye protection, and ideally with your rope tucked up in a bag) and kicking (or better yet whacking with a hammer) any protruding cobbles as you go down, to try to get a sense of how solidly they're embedded. Or, if there are smaller boulders near the base of the cliff, test those before you get out the rope.
Whether cobbles ripping out is likely or not depends almost entirely on the strength of the cement that holds the cobbles together. I've seen everything from a bomber matrix (Montserrat, Catalonia) to barely compressed dirt (The Pinnacles in California, as well as some cliffs I checked out near Murrurundi one time). With enough traffic the softer areas can become quite good for climbing (The Pinnacles is pretty popular, despite being freakishly chossy) - it's partly a question of whether what's left after all the prominent cobbles rip off is climbable.