But the best men's cashmere sweaters aren't as prohibitively ... The knit is warming, but still lightweight enough to be pulled over a button down or undershirt for a comfortable layered look.