Fighter jets were scrambled to intercept the transport planes on Monday after detecting “suspicious” activity over the South China Sea.
Malaysia’s foreign ministry described the manoeuvre as a “serious threat to national sovereignty”.
China, however, said its aircraft had abided by international law.
Malaysia, the Philippines, Brunei, Taiwan and Vietnam all contest China’s claim to almost all of the South China Sea. The row has rumbled on for decades but tensions have increased in recent years.
Beijing continues to claim an area known as the “nine-dash line” and has backed its claim with island-building and patrols, expanding its military presence while maintaining that its intentions are peaceful.
According to the Malaysian air force, the Chinese aircraft were “flying in tactical formation” at up to 27,000 ft (8.2km) and came within 60 nautical miles (110km) of Sarawak, a Malaysian state on the island of Borneo. Attempts to contact them went unheeded, a statement said.
Malaysian fighter jets were then scrambled from the Labuan Air Base to carry out “visual identification”.
Foreign Minister Hishammuddin Hussein said the planes had entered the country’s “maritime zone” and that a complaint would be lodged with Beijing.
He said the Chinese ambassador was being summoned to explain the “breach of the Malaysian airspace and sovereignty”.
“Malaysia’s stand is clear – having friendly diplomatic relations with any countries does not mean that we will compromise on our national security,” he said in a statement.
A statement from the Chinese embassy in Kuala Lumpur played down the incident, saying Chinese military aircraft enjoyed “the freedom of overflight in the relevant airspace”.
“During this training, the Chinese military aircraft strictly abided by the relevant international law and did not enter the territorial airspace of any other country,” a spokesman said.
“China and Malaysia are friendly neighbours, and China is willing to continue bilateral friendly consultations with Malaysia to jointly maintain regional peace and stability.”
In recent years the South China Sea has become a flashpoint between China and other nations in the region which claim sovereignty over two largely uninhabited island chains, the Paracels and the Spratlys.
China claims the largest portion of territory, saying its rights go back centuries.
In March, the Philippines raised a complaint with China, saying more than 200 Chinese fishing boats were encroaching on its territorial waters.
The Philippines says the boats do not appear to be fishing and are crewed by Chinese maritime militia. China has largely ignored the protest.