Why is New York not considered to be part of New England?
Because it isn't part of New England.
New England is made up of the states that were founded by religious immigrants fleeing persecution of the British crown, initially to Massachusetts. Later on Massachusetts gave birth to the colonies and states of Connecticut, Rhode Island, New Hampshire, Maine, and Vermont, at various times and for various reasons, but the bulk of settlers for these areas came out of Massachusetts, and this region has a unique, identifiable cultural heritage.
Much of Upstate New York was settled by New Englanders, and as such there is a shared cultural heritage throughout the region. Within that, however, New York City was never included in this cultural sphere, and its more diverse influences have affected the trajectory of the state at large. Moreover, the state's power center has always been Downstate, an area less influenced by New England. This map from American Nations can give you an idea of what I'm talking about:
John Smith (of Pocahantas fame) named the region New England during exploration in 1614 and on his map of 1616, partly to affirm British control of a region snug between erstwhile enemy colonies of French and Dutch, and partly to advertise himself as a mercenary in search of adventure. New York was not included because it was already claimed.
The year 1614 is particularly important as the Netherlands officially opened up New World exploration to any Dutch person who cared to undertake it (General Charter for Those who Discover Any New Passages, Havens, Countries, or Places). Henry Hudson had already explored up the river that bears his name, and in 1614 Adrian Block named the region New Netherland after returning home from a detailed exploration of coastline between New France and British-claimed Virginia.
John Smith's map helped establish British ownership of a crucial stretch of North American coastline, and he named it New England to drive home the point. That's the whole story. New York was already Dutch, all the way up the Hudson River. And while it was included under a centralized government called the Dominion of New England in America for three years (1686–1689) as others pointed out, it has never formed part of New England.
Thanks for the A2A anon. Mostly, it's because New York is large enough and populous enough that it has its own American subculture. Being the state containing "the greatest city on earth" will do that to you. There's also the historical background, that modern New York was originally New Amsterdam, claimed by the Netherlands. The land north of it was claimed by England, thus, New England. New York wasn't part of the original New England, so it's not counted as such.
There are a few who do count it as part of New England, but for the most part, Americans do not.
Because it isn't. Also, New York was originally settled by the Dutch (New York City was once New Amsterdam), and the states that are now New England were settled originally by the English. The English eventually won the colonies, so that's probably where the misconception came from.
Traditional roots are different in New York, versus New England.
However, the real bottom line is this: New York is not a part of New England because New York simply is not and has not been considered to be a part of New England. Why is today irrelevant since it is a fact. New York is also not considered a part of Pennsylvania or Ohio. Well, why not? Because that's the way it is.