No, caution is definitely the safer policy.

[Says the man in a blatantly uncautious place wiggling a large and obviously magical crystal out of the wall. Come on...]

And I can't imagine why not, though taking too many might disrupt whatever passes for an ecosystem here. This is the largest one I've tried to take.

[And... There we go! just a little more wiggling to pull it free, but there's a sound of something mineral snapping and it definitely gives a bit more.]