If thread engagement and bolt grade are the same, I don’t see why you can’t?
The square nut likely helped keep it from spinning when the bolt was being tightened.

How would I tighten the 6 point nut?
Jam a flat blade screwdriver in the gap and crank her down.
That said, I am not always right. And I don’t always do things right.

The warn manual only calls for M10x30 minimum 8.8 grade studs mounting bottom studs for warn evo 10s. So I at least match those specs except for the nut being hex instead of square. After a million trips to Lowe’s , Home Depot , tractor supply, hardware stores, I gave in on finding a square nut so I used to hex instead.

I just installed it and I realized it may be to help it stay in place as the hex was very difficult to line up until the stud caught the threads during tightening.

I believe you are correct as that’s exactly what I did to stop the hex from spinning and I believe that’s what the square nuts purpose is since the specs are the same as what the warn manual calls for M10x30 8.8 grade

Square Nuts at Empire Bolt & Screw