That line about staring at the wall clock for 45 minutes hit me hard because I did the exact same thing at a Thanksgiving years back. My uncle kept making jokes about my weight and my job in front of everyone and I just sat there frozen. What worked for me was skipping the sit down dinners at my aunt's house and only showing up to the restaurant ones, same as you. It's not too harsh at all, blocking him was the right move and you don't owe anyone a front row seat to your own bad time. If people ask why you're at the other end of the table just say you like the extra elbow room and leave it at that, no need to explain yourself.