The sense of "me" falls away when one is truly happy/joyous as a result of total involvement/presence in the here now and one is out of the snare of the sense of (psychological) time so one might find it "trivial" or "fleeting" perception-wise (coz the person-perceiver-recorder of time itself was absent)..
as opposed to sadness or (rather when) suffering (the sadness) that adds to the heaviness of the (existing) center that is "me" (aka the conditioned self/memory) which creates a sense of time owing to the underlying resistance to the associated feeling. (naturally all suffering is a result of identification with something which essentially is mental in nature. technically it's resistance to the underlying feeling which when given complete space and attention would end the mental association with the image/thought and this process of emotional stuckness with psychological suffering will reduce and end).
Without awareness suffering/sadness might feel too "person-al" and therefore "deep" as one goes onto cling to it mentally because letting it be which means feeling it fully (no matter how many times you have to keep coming bcak to it since initially you will habitually keep resisting) for it to actually go/dissipate would also mean dropping the known/the sense of (false) self the "me" it creates/is associated with it and so one passionately resists. The very act of resisting is identification/the ego/the me/the known etc etc. It eventually falls away no matter the attempts at pushing it away just that you either do it happily or life does it for you.