Today, the picturesque Port of Yarmouth mainly serves ferries bound for Lymington on the mainland. Yarmouth was the first Island settlement to be granted a royal town charter, in 1135. The town was sacked by the French in 1377 and 1524, and suffered numerous raids until Henry VIII improved the security situation by building a stone castle, in the town, which was completed in 1547. The castle is now managed by English Heritage.

An impressive feature of the town is the unique 700ft long timber pier dating from 1876, which is a Grade II listed building. Five hundred and 52 deck planks record the names of those who helped fund the pier's restoration in the 1990s.

The annual Yarmouth Old Gaffers Festival which is held in late May and early June is one of the largest events held on the Isle of Wight. More than 100 Old Gaffers - gaff rigged boats - participate in the event which is complimented by shore based entertainment. Yarmouth harbour has berthing and on-shore facilities for visiting yachtsmen.

Yarmouth Carnival, which is held in the Town Square every year in August, includes a procession throughout the town with floats, music and parades. The Yar estuary attracts walkers and bird watchers and there is a cycle way to Freshwater, along the route of the former railway line. The Hamstead walking trail runs for seven miles from Yarmouth to Brook across the Downs to the South, starting at Hamstead Ledge.