"And after the deaths of their barons, you pursued widowed ladies such as my lady Baret, and as a tyrant you had her beaten by your mercenaries [or rascals, or menials: ribaldes]** and shamefully had her arms and legs broken against the order of chivalry and contrary to law and reason, by which the good lady is forever more driven mad and lost [la bone dame est touz iours afole et perdue]."
[** that part is often mistranslated as 'making her the butt of his ribaldry']
This horrible accusation is frequently repeated as certain fact in secondary sources, and often used as evidence that Hugh Despenser was some kind of violent abusive misogynist and sadist who therefore thoroughly deserved his drawn-out and excruciatingly painful death. (The notion that Hugh raped Queen Isabella, which is purely an invention of her modern biographers, is also sometimes repeated in this context, as is Hugh's reprehensible imprisonment of the earl of Pembroke's niece Elizabeth Comyn, which did indeed happen - though there is nothing to suggest that Elizabeth was physically ill-treated, and Hugh's aim was to coerce her into signing over part of her large inheritance to him and marrying his eldest son.)
Did Hugh indeed order this inhuman and astonishingly brutal treatment of Lady Baret? Who was Lady Baret anyway? Natalie Fryde's The Tyranny and Fall of Edward II 1321-1326 - which states as certain fact (p. 117) that the lady was tortured and driven out of her mind by Hugh - identifies her as "probably the widow of Stephen Baret of Swansea." This is most likely correct. Fryde says "We do not know to what end these injuries were perpetrated," though a desire to take over her or her late husband's lands seems by far the likeliest reason, given what we know of the Despensers. This page states "The Lady Baret, widow of a Knight who fought against the crown at the battle of Boroughbridge was tortured and all her limbs broken before she gave up her lands to him [Hugh Despenser the Younger]."
Sir Stephen Baret was, almost certainly, executed with other Contrariants in the spring of 1322, most probably in Swansea. Only three chronicles mention his execution, and he was not, for some reason, named in the November 1326 judgement on the younger Despenser, as the other men executed in 1322 were. An entry on the Patent Roll of 28 April 1322, however, is a commission to four men (one of them Sir John Inge, a close associate of Hugh Despenser) to "render judgment upon Stephen Baret, a traitor, at Swaneseye [Swansea]."  Other men named in these commissions to receive judgement were all executed.
The Lady Baret supposedly tortured on Despenser's orders was most probably Stephen's widow (though might, perhaps, have been his mother). The couple had no children: Stephen's heir was his brother David, who in February 1327 petitioned the young Edward III for the restoration of his inheritance. A clerk named Stephen Baret, perhaps David's son, who was one of the guardians of the "temporalities of the bishopric of Worcester" and the attorney of the bishop of St David's, is mentioned in February 1327 and March 1334.  Stephen (the Contrariant of 1322) held lands on the Gower peninsula in South Wales, which Edward II granted to the younger Despenser in October 1320, to the huge annoyance of the Marcher lords (it led to the Despenser War the following May). According to the inquisition post mortem of Sir Guy Brian in August 1307, Stephen had "1 carucate land called Cralond" in Carmarthenshire, for which he paid a pound of wax and twelve pence annually.  A man named Richard Wroth was sent to arrest Stephen Baret in Gower on 16 February 1322, though in fact he was taken prisoner in Yorkshire sometime after the battle of Boroughbridge on 16 March by the constable of Knaresborough Castle (and must have been among those who threw away all his possessions in an attempt to flee, as he was "taken bare)".  According to a c. 1322 petition by the people of the Yorkshire village of Laughton-en-le-Morthen, Stephen had gone to the village in late 1321 or thereabouts with John, Lord Mowbray and Sir Jocelyn Deyville (both also executed in 1322) with eighty men at arms and four hundred foot soldiers. The men robbed the village and its church of livestock and goods, and took them all to the Isle of Axholme, which belonged to Mowbray. 
I have been unable to discover anything very much about Stephen Baret's wife, except that her name was Joan de Gynes and she inherited three manors in Leicestershire, Suffolk and Staffordshire from her mother Isabel de Mandeville.  According to an inquisition of July 1324, these three manors - named as Moteshale, Dadelyngton and Herliston - were then in Edward II's hands, not Hugh Despenser's.  It is unclear from the entry whether Joan was still alive at the time of the inquisition; it begins "Stephen Baret, sometime knight, and Joan late his wife, on the day of his forfeiture, jointly held the manor of Moteshale...". If Hugh did indeed have Joan tortured for her lands, he didn't hold them for long. It doesn't seem likely that he had her tortured to gain Stephen's minor holdings on the Gower peninsula, either.
What is significant is that the record of the charges against Hugh Despenser the Younger in November 1326 is the only source for the claim that he had Lady Baret tortured - and the charges against him are, as Professor May McKisack once so eloquently put it, "an ingenious tissue of facts and fiction," with a strong emphasis on 'fiction'. No fourteenth-century chronicler mentions the alleged torture. There are no petitions or commissions or inquisitions or anything else to confirm that it ever happened. The charge perhaps sounds too specific to have been completely invented, yet it is extremely odd that neither Joan - if she was still alive - nor any of her family or friends later petitioned Edward III for restitution, and even stranger that no contemporary or later chronicler noticed such a horrific act. They might have ignored the torture of a lowborn woman, but never, surely, a highborn one. I'd expect to see indignant and horrified condemnations of such brutality against a defenceless noble widow somewhere, but there's nothing. Whatever happened between Despenser and Joan Baret, the story of her broken limbs and insanity is likely to be, at best, a gross exaggeration at a time when all the ills of the 1320s were being heaped on one man's head. Whatever wrongs Hugh committed, and it's undeniable that he committed many, it seems rather unfair to assume that the story of his torturing a woman into insanity is certain gospel truth when it only appears in one document containing dubious, and some laughably inaccurate, accusations against a detested royal favourite.
1) Calendar of Patent Rolls 1321-1324, p. 149.
2) Calendar of Close Rolls 1327-1330, pp. 25, 61; Cal Pat Rolls 1327-1330, p. 14; Calendar of Fine Rolls 1327-1337, p. 319.
3) Calendar of Inquisitions Post Mortem 1307-1327, p. 32.
4) Cal Pat Rolls 1321-1324, p. 77; Calendar of Inquisitions Miscellaneous 1308-1348, p. 131.
5) The National Archives SC 8/7/301.
6) Cal Inq Misc, pp. 200-201.