Unsupervised signal modulation clustering is becoming increasingly important due to its application in the dynamic spectrum access process of 5G wireless communication and threat detection at the physical layer of Internet-of-Things. The need for better clustering results makes it a challenge to avoid feature drift and improve feature separability. This paper proposes a novel separable loss function to address the issue. Besides, the high-level semantic properties of modulation types make it difficult for networks to extract their features. An autoencoder structure based on the Random Fourier Feature (RffAe) is proposed to simulate the demodulation process of unknown signals. Combined with the separable loss of RffAe (RffAe-S), it has excellent feature extraction ability. Great experiments were carried out on RADIOML 2016.10A and RADIOML 2016.10B. Experimental evaluations on these datasets show that our approach RffAe-S achieves state-of the-art results compared to classical and the most relevant deep clustering methods.