The spiral core instability, observed in large aspect ratio Rayleigh-Benard
convection, is studied numerically in the framework of the Swift-Hohenberg
equation coupled to a large-scale flow. It is shown that the instability leads
to non-trivial core dynamics and is driven by the self-generated
vorticity. Moreover, the recently reported transition from spirals to hexagons
near the core is shown to occur only in the presence of a non-variational
nonlinearity, and is triggered by the spiral core instability.
Qualitative agreement between the simulations and the experiments is
demonstrated.