When John Macarthur died in 1836, his Pyrmont estate was subdivided.  Thomas Chowne bought land at Elizabeth Macarthur Bay in 1840, and began to build ships. This venture attracted other processes (such as Fyfe’s iron works) that would otherwise disturb the citizens of Sydney. A wooden bridge linked Glebe Island to Pyrmont in 1857, bringing cattle to (and carcasses from) Glebe Island abattoirs.
City Iron Works from 1865 added to the noise and stench. By 1871 the Works burned 2,500 tons of coal each year:

The City Iron Works employ sixty men. Scrap iron is worked up here and pig iron is puddled [in furnaces, to produce wrought iron]. About 100 tons a month is rolled here, and railway wheels and tires for contractors’ locomotives have also been turned out.

Inland, stone was quarried. Saunders’s quarries were more famous, but O’Brien’s ballast quarry had a more immediate effect here, flattening the land now occupied by Cadi Park and Reflections, creating the escarpment which gives the building its name. After the 1850s gold rush however, Pyrmont yellow block sandstone was in huge demand for the impressive public buildings that transformed the face of Sydney.

CSR bought land around Elizabeth Macarthur Bay for a refinery that opened in 1878. Sugar prices collapsed in the 1880s, and CSR’s survival was threatened, but the company survived to achieve a virtual monopoly of sugar production in the South Pacific. Raw sugar arriving by ship was stored near the refinery (where Regatta Wharf and Fleetview now stand). The disused ballast quarry was ready-made for storing raw sugar: its rock walls only needed roofing. Ships moored nearby to unload sugar; men wrestled sacks onto drays, and horses hauled the drays to the Raw Sugar Store.
Over time this process was mechanised and streamlined. By the 1950s sugar was unloaded by high-powered cannon. 140lb raw sugar sacks were slit at the ‘cutting in’ station and the contents tipped into a shaker conveyor – upended with a sack hook (a rookie often fell in).

As CSR produced almost all Australia’s sugar, it was essential to hold large reserves of raw sugar so that production was never interrupted. To meet this demand, the vacant quarry grew into a series of fully enclosed, high-roofed sheds stretching from the wharf as far inland as the Tablet House and Cooperage.