as Rowan says, depends... send us a photo and I have no doubt you will get several differing opinions.
Crystals or minerals on matrix tend to be more sought after (not always: hence the qualifier "depends") as the matrix can help nail down or narrow down the locality of the piece.
Esthetics matter. Does the matrix enhance the overall look of the specimen?
If you are looking to re-sell the material, matrix can help increase the specimen size from TN to miniature to cabinet piece if the proportions are right.
Matrix can provide a contrast to the main crystal as well as a natural pedestal for the crystal.
But sometimes, I think, a single crystal does just fine all on its own.
If you are uncertain about taking the matrix off, leave it be - this is one bell you cannot un-ring!
Maggie