Guilt cannot be imputed to someone who is not aware that the act is gravely sinful. If he was given proper instruction then, considering that he contradicted his father, it probably hasn't made much of an impression. Because he has been told masturbation is wrong does not necessarily mean he knows and believes this to be so. He's a kid so he probably doesn't even think about this stuff.
I'm not sure what kind of God you believe in Rosarium, but mine doesn't send 12 year old boys to hell for masturbating.
I usually do not join in on these kinds of discussions because of their heated nature. But I feel compelled to add something.
Awhile after me "reversion," if you will, God gave me the grace to realize that there is a big difference between knowing
something is a sin and believing
something is a sin. I had been taught what was right and what was wrong, I just refused to believe
certain things were wrong due to the liberal environment around me. At one point, all those sins that I knew were wrong but refused to believe were wrong came flooding back to me and I was abhorred at their gravity. Needless to say I headed ASAP to Confession. My wonderful Confessor said, "Isn't God wonderful that he gives us time before these realizations overtake us so as not to discourage us when we are just returning to Him."
My point is, the son in the OP knows his actions are wrong. He is above the Age of Reason and his father has instructed him. The son is being obstinate in his unbelief.
Suppose the situation was changed. Suppose the son is standing on the railroad tracks and a train is barreling toward him. The Dad says get off the tracks or you will be killed. The son now knows he will be killed, but says, I don't believe it, so I think I'll just stay where I am. When the train hits he will be dead whether he believed it or not.
So,if the son in the OP continues in his sin, he is culpable as his father has instructed him that it is serious matter and he does, indeed , know, whether he believes or not.
Now I am not God and cannot judge an individual soul. But the Dad in the OP and I both know that God is all just as well as all merciful. And it is better to spend this life making reparation to His justice so that God will be merciful to us at our judgment.
Just as a good father would make every attempt to remove his son from the train tracks before the son suffers from the consequences of his non-belief, the OP Dad wishes to make sure his son does not suffer the supernatural consequences of his un-belief.
The only advice to that dad that I can offer over and above what has already been said, is to pray constantly for one's children. I'm sure it was my mother's prayers for me that I think will eventually aid in my salvation.