So my store is hosting a small tourney with the Battle of Yavin rules.

I'm inclined to play Rebels and go with quantity over quality. Specifically I'm thinking about playing either 4x Blue Squadron Escorts w/Protorps, or 2 Y's and 3 X's. I've had a lot of success with 5-ship rebel lists recently, but I usually sub one of the X's for a U-Wing, which won't be legal. I like flying in formation... fragile aces might be able to dodge some of my arcs, but not all of them. In my games, I've had good luck in attrition battles against fewer, higher-initiative ships. I also figure that there will be plenty of TIE swarms and having a swarm of my own might increase my chances of surviving the withering hail of fire.

Question is, if I do the 3X/2Y, should I put R4 Astros and Dorsal Turret on the Y-Wings, or should I drop the Astros and upgrade the turrets to Ions? Or is it way better to have 4 X's with Proton Torps? Or should I scrap the idea entirely and go for something else?

You could also go with Biggs and a trio of Red Vets with R4s & Selfless. It lacks the punch of torps, but it’ll spread damage across the formation and make the most of your durability.

I do like the idea of spreading damage to survive initial contact with the enemy, while using my own 4x 3-dice shots to remove an enemy ship without losing my own. I think B-Wings are probably better for this (with their higher shield ratings) but it should still work. Pity there aren't enough points to put R5 Astros on the ships to help cancel the crits sustained with Selfless.

Have you tried this squad yourself?

I have not, but saw a similar squad using four B-Wings and Biggs. I think the Gold Squadron podcast has it on their YouTube channel.

This is one I'm looking at running in Hyperspace events, -and fits your rules:

Hyperspace 1.3

(52) Wedge Antilles
(0) Servomotor S-foils
(3) Swarm Tactics
Points 55

(43) Red Squadron Veteran
(0) Servomotor S-foils
(3) Swarm Tactics
Points 46

(43) Red Squadron Veteran
(0) Servomotor S-foils
(3) Swarm Tactics
Points 46

(48) Biggs Darklighter
(0) Servomotor S-foils
(5) R5 Astromech
Points 53

Total points: 200

now I'm unsure how it will perform, but being able to spread some damage (with Biggs) is good, plus all 4 ships shooting at i6 (you'll always want to be player 1 against Aces) could be very potent