Overcoming opponent's magic
If Natsu can burn away the time-reversing magic of someone with infinite power, what makes you think he wouldn't be able to burn away the magic behind Gildarts' crash? Natsu can burn away Zeref's black magic, which can put a hole through Larcade. The strongest attacks crush has managed to break is God the historia of Serena's attacks. Not only was the historia said to be weaker than the actual living God Serena, but Natsu actually has stronger attacks than God Serena himself when he was alive. Natsu's flames have overcome stronger magic and greater hax than Gildart's crush.
Attack Power
Even without accounting for the flames of savage emotions, which I remind you isn't actually restricted, Natsu's demolition fist when performed in Dragon Force was able to break down Zeref's physical body. Zeref was able to recover from this because he was immortal and by reversing time. Nobody else possess immortality to remain alive after being scattered, and a means of putting themselves back together by reversing time. Now Zeref can actually withstand attacks from Igneel's power without his body breaking down, so the power behind this attack must be great. At the very least, it would still very likely be able to overpower Empyrean and deal enough damage to knock out Gildarts.
Durability/Endurance
With the exception of Ars Magia, which Gildarts did not really tank, none of the attacks Gildarts has received outclassed the attacks that Natsu has endured. Irene enchanted Neinhart to be stronger than the limits of his own magic, and the historia of God Serena being Neinhart's magic. This means that God Serena's winds were in all likelihood stronger than historia God Serena's attacks. The feats support this as Neinharts winds actually managed to cut up Natsu and Brandish, whereas God Serena's attacks didn't really do any noticeable damage. Zeref's black flames blew a hole in Larcade, so it's on an even higher level than that. And of course, Natsu managed to survive against August's nuke, which was August's most impressive attack after Ars Magia.
Speed/Agility
The main argument for Gildarts' speed seems to be that he can tag August a few times, and that August was faster against Jelall's group. Now I don't think August is slow by any means, but Augusts' magic is to copy the powers of his opponents. When August fought Jellal, Cobra, and Racer, he could copy meteor, slowing magic, and cobra's hearing. Each of these factors grants August a significant edge in terms of agility and speed. August did not have these powers when he faced Gildarts and Cana (or if he did, Hiro never made it clear). I will give Gildarts credit where credit is due for getting the drop on August, as August is still a high level Spriggan. However, even Cana was able to tag August with her cards. Gildarts was also not consistently faster than August. Towards the latter half of the fight, August was tossing Gildarts around in hand to hand.
Why speed is not the dominant factor in this match
Speed was never the dominant factor in Natsu and Gildart's previous sparring sessions. Speed was never the dominant factor in any of Gildarts' previous victories (against Bluenote and historia of God Serena). Even when Natsu was weaker than Gildarts, he could still find ways to force Gildarts into taking one of his blows. Also Natsu doesn't need to tag Gildarts a bunch of times. He could just try to clash fists with Gildarts, and do the exact same thing he did to Zeref. If Gildarts uses Empyrean, then Natsu can just throw demolition fist at the exact same.
Hand to Hand/Combat Experience
Now I am willing to say that Gildarts has the edge in hand to hand combat because of his older age, and that means he probably has spent a longer time training his hand to hand combat. However, does Gildarts actually have more experience? Natsu has fought all the leaders of the Ballam Alliance (Zero, Hades, and Mard Geer), a dragon (two if you count dragon cry), two actual Spriggan 12 members (one of which is enchanted), and an immortal with infinite power that can reverse space-time. Even if Gildarts has the advantage in hand to hand combat skills, Natsu's experience in combat isn't so inadequate that he wouldn't be able to keep up.