Part of a message board post I wrote to a mom new to homeschooling and to unschooling, dealing with frustration that her child is not the slightest bit interested in anything that resembles "work" or "schoolwork"...
Unschooling does not mean giving your children educational autonomy in the expectation that they will willingly gravitate to exactly the sorts of things you would have required of them in the first place. As uncomfortable as it may seem, for unschooling to work properly you need to make peace with the possibility that your child may not choose what you want him to choose -- and he needs to know that he will be valued and supported in those 'other' choices. I'm not saying this is easy! It is something I continue to wrestle with, even ten years into this racket.
It's important to realize that just because your child is making 'other' choices doesn't mean that unschooling isn't "working." Sometimes the learning is just going in different directions, or is quiescent, or is building momentum under the surface, or doesn't look like what we expect. It's going on all the same. Sometimes the learning is more holistic but more important than we can see when we're in the thick of it. My eldest spent a few months doing almost no violin playing at all ... and what she learned was that she missed it, and that she has a passion for it -- and that knowledge has propelled her to fantastic heights in the year and a half since. Another example: I used to worry quietly to myself about my son who had almost no interest in writing and spent all his time on the computer tinkering with games and code. He even began talking in internet acronyms, saying ROFL (pronouncing it aloud as "rawful") rather than laughing. But then one day I realized that he was not only learning complex physics through game programming, but he was reading bits of German (game developers' documentation) and writing some pretty cool stories and reviews in a computer-gaming vein. I was so busy worrying over what he wasn't doing that I didn't see all the learning that was going on.
So my first two pieces of advice would be to try to let go of the expectation that your child will naturally gravitate to exactly the learning that you'd have chosen for him, and to hone your observation skills so that you can see the learning that is actually going on instead.
My other advice would be to put your relationship first. Don't fuss with schooling, outcomes or academic expectations at all until you've sorted out your family relationships. It sounds to me like there's a lot he's doing behaviourally that is pushing your buttons, and that there's an atmosphere in your home of frustration and disappointment and that Some People may not be living up to others' expectations of them. Ultimatums in disguise, unhappiness and all that.
That's the first front I'd work on. I've always found that happy children who like their lives and like being with the people they love are hard-working, optimistic, motivated children who enjoy making their parents happy. I've felt like you do more times than I'd care to admit. I'd say that 99% of my despair is solved by a change in my attitude that then changes my relationship with my child. Not only do I view things-as-they-are more favourably, but my child behaves more favourably and things actually improve in a real sense.