In October 2018, genealogist Douglas Richardson made the exciting discovery that in 1329, Roger Damory had two surviving legitimate daughters and heirs: Margaret and Elizabeth. See his post in soc.genealogy.medieval, and here is an image of the original document that names the two Damory daughters. Margaret Damory is named before Elizabeth, and therefore was probably the elder. It would seem, therefore, that Margaret Damory was still alive in 1329 but died sometime between then and 1337, when Elizabeth Bardolf was first described as Roger Damory's sole heir. It might not be a coincidence that Elizabeth is called Roger's sole heir for the first time, as far as I can tell, in the charter rolls and the patent rolls on 8 and 13 August 1337. Perhaps Margaret had recently died.  Elizabeth Damory married John Bardolf before 25 December 1327, and their son William Bardolf was born in Wormegay, Norfolk on 21 October 1349.  They had daughters Agnes and Isabel as well, who are mentioned in their grandmother Elizabeth de Burgh's will of 1355 but appear to have died young or joined a convent.
The question is, who were Nicholas, Richard and William Damory/Pour(e), the Cambridge students? This Richard is most unlikely to have been the Sir Richard Damory of Oxfordshire born c. the 1270s; he was far too busy acting as the guardian of the future king of England in the late 1310s to be a student at Cambridge. Richard's son the younger Richard was only born c. 1315, so obviously it can't be him either. As well as his decades-long association with Roger Damory's widow Elizabeth de Burgh, Nicholas Damory was associated with Elizabeth Damory Bardolf: she and her husband John Bardolf granted him the Oxfordshire manor of Holton, which she had inherited from her father Roger, for life in 1340, and John appointed Nicholas as his attorney when he went overseas in 1363.  The Complete Peerage speculates that Nicholas was probably a first cousin of Richard Damory the younger (c. 1315-1375).  If this is the case, he was the son of either Roger Damory, or of another Damory brother whose existence has never been discovered. Might he have been an illegitimate son of Sir Roger Damory, and thus Elizabeth Damory Bardolf's half-brother? He can't have been Roger's legitimate son from a putative first marriage before he wed Elizabeth de Burgh in 1317, as otherwise Nicholas would have been his heir.