There are people who willingly label themselves as part of a group: straight, gay, queer, white, POC, multiracial, etc.
There are people who, when asked how they identify, casually say they don’t use labels. Or perhaps, they understand the use of labels but think we’d all be better off without them.
Then there are people who, when they see certain labels, are sent into an uproar. ‘Labels are divisive! We’re all the same! What’s really important is how you treat other people!’

I can agree with the importance of equality and good treatment. The problem with ignoring labels is it also tends to ignore differences. The ‘default’ person, at least in modern Western society, is a very specific type of person: cis, straight, white, able-bodied, and male, among other things. Obviously, not everyone matches this description, but it’s what most people assume unless explicitly (and, often, repeatedly) told otherwise. Look at the recent uproar over Rue being portrayed by a black actress in the Hunger Games: despite being described (albeit briefly) as black in the book, most people assumed she was white, because white is default. I do this as well; I noticed only Rue’s dark eyes, and went on to presume she was white. I assume all the characters in another book I’m currently reading are white, too, though come to think of it I can’t actually remember the descriptions of any of them, save one girl having dark hair.
In a way, these assumptions are understandable, since the majority of people are straight, cis, and able-bodied (and disproportionately in most media where we can see them, white and male), but ignoring the existence of anyone else can quickly lead to a complete lack of knowledge about them. When it comes to invisible characteristics, like sexual orientation or gender identity, it can lead to not even realising certain groups exist.

Of course, labels are not perfect things. There are problems with them, such as people trying to shove labels onto other people, sometimes quite inaccurately. The assumption anyone in a same-sex relationship is gay or a different-sex relationship is straight has irritated many a bisexual/pansexual/asexual person. Even worse is when a person is presumed to be gay or trans* because they act in a non-heteronormative fashion, or when an LGBT person is presumed heteronormative because of their behaviour or appearance. (‘She’s got short hair, so she’s a lesbian, but this one – she’s too pretty to be gay!’)
On the flip side, there are people who fit all the ‘requirements’ for a certain label, but don’t actually identify with it, or identify more closely with something similar, but not quite the same – for example, there are people who could be considered ‘transgender’ but consider the difference between their gender designation at birth and their actual gender identity to be more a medical problem than an identity issue, or people who would technically be called ‘homoromantic asexuals’ but find it easier to describe themselves as simply ‘gay’. It’s impossible to force anyone to apply a certain label to themselves, and repeatedly referring to them with a label they don’t like will only annoy them.

There’s also the problem of people who use different labels automatically degrading anyone who doesn’t fit ‘their’ label. As an example, a prominent gay/lesbian attitude toward bisexuality is to view it as ‘a term gay guys in high school use when they want to hold hands with girls and feel like a normal person for a change’ (thanks, supposedly diversity-friendly Glee) or ‘straight girls trying to get male attention’. There’s an entire hierarchy in the trans* community: those going stealth above those who aren’t yet passing above those who don’t even want surgery above those who aren’t even binary above those who prefer another gender’s clothing but don’t actually identify as another gender… Using the word ‘bisexual’ or ‘genderqueer’ can thus result in getting a fair amount of hatred not only from anti-gay and anti-trans* people, but from other non-heteronormative folk as well. The vast majority of people I’ve seen claiming they ‘don’t use labels’ are cis and straight, but I can’t exactly blame those who don’t fit the standard LGBT acronym (and even some who do) from avoiding otherwise appropriate labels and their accompanying mud-slinging. It’s hard enough having our enemies describe as as sinful perverts without people who should be our allies claiming we don’t really know who we are on top of it, and it leads to further internal conflict as bisexual people try to figure out if they’d rather identify as ‘gay’ or ‘straight’ and transmasculine DFABs try to figure out if they are ‘manly’ enough to qualify as ‘men’.

There’s no real culture-wide solution to stopping one person from forcing labels on another, or the group of people clumped under one label from discriminating against the group of people under another. On an individual basis, we can only use the labels we’re most comfortable with (even if we’re comfortable with none at all) and accept the labels other people are comfortable with (even when they lay under a complicated tangle of overlapping and sometimes seemingly incompatible terms).

Further reading

Splitting Hairs: Is the LGBT Community Alienating Its Own?