Ground-state phase diagrams of the anisotropically interacting Kitaev-\Gamma-Heisenberg models on honeycomb lattices

Kitaev materials have attracted much attention because of their novel properties. Honeycomb-lattice magnet \alpha-RuCl3 is a candidate material. The first-principle calculations have indicated that the symmetric-offdiagonal interaction (\Gamma) for the nearest-neighbor effective spins is the second dominant interaction in addition to the most dominant Kitaev interaction (K) [1-4]. Moreover, the Heisenberg interaction (J) may play the role in spite of its small amplitude. Thus, the model including these three interactions (the K-\Gamma-J model) on a honeycomb lattice can be a minimal model for \alpha-RuCl3. The K-\Gamma-J model itself is known to show various phases depending on K, \Gamma, and J [5]. In particular, it has been argued that a spin liquid state distinguished from the Kitaev spin liquid appears in strong \Gamma interaction [6]. In spite of the intensive studies, the details of the whole phase diagram of the Kitaev-\Gamma-J model have been left unclear.
In this study, we investigate the ground-state phase diagrams of the models with K and \Gamma interactions (the K-\Gamma model) and with \Gamma and J interactions (the \Gamma-J model) on a honeycomb lattice by including the effects of the spatial anisotropy of the interactions [7,8]. In fact, the first-principle calculations have shown that K, \Gamma, and J interactions have the spatial anisotropy. We adopt two approaches: (i) By starting from the isolated dimer system, the interdimer interactions are increased to the isotropically interacting system. (ii) By Start from the spin-chain system, the interchain interactions are increased to the isotropically interacting system. In the former approach, we investigate the stability of the dimer state by combining series expansions and numerical exact diagonalizations (ED). In the latter approach, we first investigate the stable states in the spin-chain system using density-matrix-renormalization-group calculations and ED. We then investigate how they change, as the interchain interaction is increased using ED.
For \Gamma>|K| (K<0) in the K-\Gamma model and for \Gamma>|J| (J<0) in the \Gamma-J model, the dimer state survives up to the isotropically interacting system, where the phase transition occurs. The results attribute to the frustration effect caused by the opposite signs of \Gamma and K in the former model and \Gamma and J in the latter model. This frustration effect suppresses the transition to a magnetically long-range ordered state. Actually, in the same sign regions where the system is free from this frustration effect, the dimer state undergoes a phase transition to the magnetically ordered state in both models, before the interactions become isotropic. In the other opposite-sign regions of \Gamma<0 and K>0 in the former model and of \Gamma<0 and J>0 in the latter model, various states appear as compared with the frustration-free regions. In the spin-chain system, three kinds of states appear: Tomonaga-Luttinger liquid (TLL) states and other two states. When the interchain interaction is increased, the TLL states at K<0 and \Gamma>0 in the K-\Gamma model and at J<0 and \Gamma>0 in the \Gamma-J model connect to a two-dimensional ordered state without showing a vestige of the phase transition.
The obtained results are summarized in the two phase diagrams of the K-\Gamma and \Gamma-J models. The phase diagrams indicate that the ground state is sensitive to the spatial anisotropy of the interactions. We find that the isotropically interacting system is placed on the phase boundary in the large parameter regions with opposite sings of K and \Gamma and of \Gamma and J. This indicates that the spin liquid argued in the strong \Gamma interaction is unstable against the spatial anisotropy of the interactions.

[1] H.-S Kim and H.-Y. Kee, Phys. Rev. B 93, 155143 (2016).
[2] S. M. Winter, Y. Li, H. O. Jeschke, and R. Valenti, Phys. Rev. B 93, 214431 (2016).
[3] R. Yadav, et al., Sci. Rep. 6, 37925 (2016).
[4] C. Eichstaedt, et al., Phys. Rev. B 100, 075110 (2019).
[5] J. G. Rau, E. K.-H. Lee, and H.-Y. Kee, Phys. Rev. Lett. 112, 077204 (2014).
[6] A. Catuneanu, Y. Yamaji, G. Wachtel, Y.-B. Kim, and H.-Y. Kee, npj Quantum Materials 3, 23 (2018).
[7] T. Yamada, T. Suzuki, and S.-I. Suga, Phys. Rev. B 102, 024415 (2020)
[8] T. Suzuki, T. Yamada, and S.-I. Suga, Phys. Rev. Lett. 103, 224425 (2021).

E-mail for corresponding author: [email protected]