The two leaders reviewed the progress made on the Mumbai-Ahmedabad bullet project and signed an agreement on the implementing arrangement for deeper cooperation between the Indian Navy and the Japan Maritime Self-Defense Force (JMSDF).
Modi has called for regional unity against protectionism.
It was also an opportunity for Abe to make a pitch to Indian tourists to visit the mountain, which is considered as a national symbol and pride of Japan.
Abe and Modi are expected to issue a joint communique on strengthening security cooperation and measures aimed at pursuing a "free and open Indo-Pacific", Kyodo News agency reported on Sunday, citing an unnamed Japanese foreign ministry official.
Condemning in the strongest terms the growing threat of terrorism and its universal reach, the two leaders expressed their desire to strengthen cooperation against terrorist threats from groups including Al-Qaida, ISIS, Jaish-e-Mohammad, Lakshar-e-Tayyiba, and their affiliates.
PM Modi said Japan investors have announced they would invest $2.5 billion in India.
There was, however, no development on the US-2 amphibian aircraft, which India seeks to purchase from Japan.
By 2022, India's digital economy is expected to grow to Dollars 1 trillion and lead to creation of 10 million jobs, according to consulting firm EY.
China is a strategic and economic rival of India.
"[We] will have a frank exchange of views and I'd like to deepen our cooperative ties for peace and prosperity in the Indo-Pacific region", Abe said at a meeting of his Liberal Democratic Party on Tuesday.
The two leaders have reportedly become friends, and Modi became the first foreign leader to receive an invitation to Abe's holiday home in Yamanashi near Mount Fuji, where the pair held informal talks on Sunday.
US President Donald Trump's policies, which have targeted mostly China with tariffs, but also Japan and other nations, accusing them of unfair trade practices, are working to prod India and Japan to promote their economic ties. "I have been told that today Japan investors have announced that they will invest 2.5 billion United States dollars in India", he said at the joint briefing.
The PM also addressed the Indian diaspora in Japan during which he invited the Indian community in Japan to contribute actively in building a "new India".