It depends on where you are in the world. For instance, in France, the focus is in keeping the cows healthy enough that the milk is safe. I lived in Normandy for a while, never actually drank milk there as I lost the taste for it at 18-19, but consumed a lot of raw-milk Camembert cheese. If you've visited France, wondered why their soft cheeses are so amazing - it could be this (and no, they're pretty much not available in the US).

I wouldn't drink raw milk here in the US, unless I was on the farm, knew the 'girls', knew their conditions. Speaking of some people being allergic, there's some evidence that the breed of the cow, and specific protein variants in the milk can be an issue. Sadly, the breeds of cows developed to produce a lot of milk have a lot of this protein variant, which is called A1 as it was discovered first. Older breeds (Jersey, Guernsey, etc.) tend to not have it - but there's genetic variability so some do. That's called A2, because it was discovered second, in spite of being the originally-most-prevalent. Some people who seem to have a milk allergy - which is different from just not having lactase as adults - are actually reacting to A1. Google 'A2 milk' for a full explanation.