What a complicated question. There is no wrong answer. Each has significant advantages and disadvantages versus the other. I am a decidedly rifle guy. I always was in my youth. However, I spent a few years as a dedicated shotgun guy and I was convinced it was my go-to defensive option for that duration. I did all the training, got good at reloading it, etc… Then I went back to being a rifle guy. Therefore, I have worked a lot with each and I can at least speak to it from that angle. For myself I am entirely a handgunner first. My top training priority always goes to handgun. But to be squared away you should maintain at least one defensive long gun. Therefore, the debate between shotgun and rifle always comes up.
Now in the 21st Century the rifle, in particular the AR15 M4 carbine variant, is all the rage. I am an AR guy as well. For me there is one overriding reason that this platform is my primary long gun: versatility. The AR15 is absolutely versatile. Set one up for home defense, set one up for hunting, etc… It is very user friendly, not punishing to shoot, and aftermarket support is greatly available. Therefore, it is my choice. However, I can offer you a brief comparison between the excellent and versatile AR platform, or other carbines for that matter, and the shotgun of any modern pump action or auto variant.
Let’s begin by summing it up: the rifle offers greater range, greater accuracy, and greater ammunition capacity, has less recoil, and is substantially simpler to operate. Those are the advantages. The shotgun hits harder within its range and the spreading pattern of buckshot enhances hit potential, and thus the speed at which you can makes shots. Those are the shotgun’s advantages. The disadvantage of the rifle is that it does not offer the spreading pattern and demands more traditional marksmanship than does the shotgun within buckshot range. The shotgun has the disadvantage of being much more difficult to manipulate, more difficult to shoot due to recoil, limited in ammo capacity, and limited in range. Pick one. Bottom line: they are both vastly more capable at resolving social disputes than any handgun.
My thoughts are that the rifle remains the most versatile platform. Due to my life obligations my firearms training has limits, as it realistically does for any of us, and since my training priority goes to handgun what time and resources I have left for long gun goes towards the AR. That is my approach. It does everything reasonably well, so it is my go-to long gun. If you have the ability to train with both platforms then certainly do so, but the reality is most people spend too little time training with any of their guns and I think the average defensive shooter is better served in prioritizing one or the other. Which one to prioritize is a personal choice. Here are a few considerations if making that choice:
The shotgun is the heavy hitter at close range and if your environment and foreseen needs exists within a limited range this is something to consider. Do you live where your possible defensive needs would be typically within the reach of buckshot? Most defensive use in civilian life would be within that range, let’s face it. Also, do you live in a rural environment where critter control, like raccoons in the chicken coup, would be part of your long gun use? The shotgun is hard to beat here.
On the other hand, do you want to set up a gun that multiple members of the house may use like your small wife and teenage daughter? The shotgun can be a bit of a brute for small people, but the AR carbine with a multi-position stock is great for this. Do you live in a rural area and your critter control needs are not raccoons in the coup but Coyotes 100 yards away from the house? Perhaps two legged threats would be at farther ranges in such an environment as well. The rifle is where it is at here. Do you have emergency plans that would involve “bugging out” to leave a congested urban or suburban area? If so, you can carry four times as much 5.56 ammo for the same weight and bulk as 12 Gauge ammo, something to consider.
Ultimately it comes down to your predicted needs, your environment, your training resources, and certainly just personal preference. Either is a substantial force multiplier and setting up a defensive long gun is a sound decision.