The earliest mention of Coldharbour is in a deed in the Petworth House archives dated 1718. It records that Mrs. Mary Cockquerell Widow of George Cockquerall of Heyshott who had inherited the lordship of the manor from his father in 1699, sold the property to John Colebrook of Vining in Ambersham.
 
It was then descibed as -
the messuage, barn, gate room, garden and orchard called Coldharbour with 4 closes of land and a further 2 acres of land and furze to the west of the messuage adjoining the common. commoning right include, the digging of chalk on Heyshott Hill and laying it on the Common, to carry away, at will, turf from the common and to pasture cattle, at will, on the unenclosed Common.. Also such right as sitting in the pew near the pulpit in Heyshott church as the inhabitants of the messuage have heretofore had.
 
In 1732 the property passed to John Shudd who had inherited Berrywood Farm in trust a year earlier. It now consisted of 59 acres and remained tied to Berrywood until the 20th century.
In 1827 the 3rd earl of Egremont bought Berrywood and Coldharbour and his son George Wyndham inherited them 10 years later.
 
At this time the Coldharbour buildings were described as
house in 3 tenements, barn, hovel [cart shed], yard & garden.
 
The 3 tenements were tied for workers on Heyshott farms
In 1851 the census showed 4 families living at Coldharbour -
James Poet, farm labourer, aged 44 Sarah, his wife & 7 children.
Richard Poet, farm labourer, aged 67 Elizabeth, his wife & 3 children, plus lodger George Hopkins, 88 yr old farm labourer.
Lucy Robinson, a widow aged 50, 2 children and a visitor.
Frances Tiller, a widow aged 30 and 2 children
A total of 21 residents and one visitor.
 
There was one well to serve the 3 households, this was near No.3.
 
In 1931, John Hutchings, who was farming Manor farm in Cocking took over the lease of Coldharbour from Walter Lovejoy and he, and then his son, farmed the 81 acres until 1951 when it was let to Jack Robinson to graze horses. In 1954 the land was incorporated in Manor Farm under Peter Dalglish.
 
From the time of the 1926 exchange between the Leconfield and Cowdray estates, No.1 Coldharbour [the cottage nearest the road] ceased to be tied to a farm, but the other 2 continued as such. No.2 being tied to Berrywood and No.3 to Leggs farm.
 
The Tiller family lived for about 2 generations in No.1, the last being Charlotte Wrapson, nee Tiller. After her it was occupied by Mr and Mrs Stubbington and when he died and Mrs Stubbington went into sheltered accomodation the 3 cottages were put up for sale. The smaller number of workers on the farms meant that fewer tied cottages were needed.
There was a proposal to turn the property into a Youth hostel which led to divided opinions in the village. It was eventually put up for auction and bought by Yvonne Richards who occupied it, on her own, for about 10 years.