The city of Trincomalee presently situated in the Four Gravet Divisional Secretariat of the Trincomalee District, was known as Gokanna in the ancient times. Fort Fredrick was built encompassing the summit of Swami Rock, the highest point in the city. Swami Rock has had Hindu and Buddhist connections from the bygone times. King Mahasen (275-303 A.D.) is said to have demolished the Jaina monastery that existed on the Swami Rock and built a Buddhist temple with a preaching hall. In 1724 A.D the Portuguese general Constantine De Saa destroyed the temple to build a fortress.

The Dutch who colonized the country after the Portuguese, caused to have enlarged the fort, while the British who governed the country after the Dutch enlarged it further and named it as Fort Fredrick. Excavations conducted within the fort led to the discovery of Hindu and Buddhist sculptures of archaeological interest.

A Tamil rock inscription belonging to the 16th c. A.D. is found on the main entrance to the fort. It says that the Hindu temple that existed at this place was destroyed by the Portuguese. Sculptures of God Vishnu and Goddess Lakshmi and a Sanskrit rock inscription of 1223 A.D. which mentions of the arrival of a person known as Codaganga were discovered. Leisure Divers diving in this area of the sea had discovered submerged Buddhist and Hindu sculptures and had brought them ashore. Ruins of the ancient Gokanna monastery also had been found in the fort premises.