Myth: Edward II cruelly and deliberately punished his queen Isabella of France by taking three of her children (and they're always just 'her' children when this tale is told, never 'their' children) away from her in the the autumn of 1324 and sending them to live with other people. The story goes that at around the same time as Edward confiscated Isabella's lands in exchange for a much smaller income and removed her French servants from her household in September 1324 when he was at war with her brother Charles IV of France, he also nastily and cruelly 'stole' their three younger children John of Eltham (b. August 1316), Eleanor of Woodstock (b. June 1318) and Joan of the Tower (b. July 1321) from Isabella's care and sent them to live with Eleanor Despenser née de Clare in John's case and Isabella Hastings née Despenser in Eleanor and Joan's case, in order to punish and hurt the queen.
Given that a great deal of what is written about Edward II
nowadays as 'fact' simply melts away into nothing or turns out to be grossly exaggerated or twisted when you examine the primary
sources - so much of what is written about
Edward is just modern writers copying from other modern writers, mistakes,
myths, misconceptions and all, without bothering to check the primary
sources - I decided to look into this often-repeated story in detail. The first
things I wanted to establish were: What contemporary sources state that Edward II set up households for
his children in September 1324 or thereabouts? What source(s) state(s)
that he did so punitively with the intention of hurting Queen Isabella by
deliberately and cruelly removing her children from her? And most
fundamentally, how do we even know the children were living with Isabella as their sole or primary carer in
the first place, given that it seems
most odd for a fourteenth-century queen to have been the full-time carer of her children as the 'they were cruelly taken away from her' story seems to imply?
The notion that Edward II 'removed' Isabella's
children from her as a punitive action at about the same time that he confiscated her lands in
September 1324 is a discovery, or rather an invention, of the writer Paul Doherty in his 1977 Oxford doctoral thesis about Isabella, repeated in his error-strewn 2003 book Isabella and the Strange Death of Edward II.
 No other writer, in non-fiction or fiction, mentioned it at all before then, but since then the story has become
grist to the mill for followers of the currently trendy Victim!Isabella school of
thought. One fairly recent self-published novel includes a
heart-rending and heavily foreshadowed - Isabella talks constantly
throughout the book about how precious her children are to her and how it would
destroy her to lose them - scene where Isabella's little daughter Joan is torn right out of her arms and her son John slapped across the face for trying to
resist the nasty evil cruel men coming to remove them from her on nasty
evil cruel Hugh Despenser the Younger's orders, with nasty evil cruel Edward II approving this behaviour towards his own children. Oy vey. This novel, in addition to having Isabella talk frequently about how much she adores their children, has Edward II so indifferent towards them he struggles even to remember their names. As a way of portraying Isabella as a likeable character to readers and her husband as unlikeable - and oh my, is Edward meant to be unlikeable here - it's about as subtle as being bashed over the head repeatedly with a sack full of sledgehammers.
For his allegedly factual statement that Edward removed Isabella's
children (as though they weren't his children as well, for pity's sake) from her in or after September 1324 in the interests of "[e]ven greater
cruelty" towards her, i.e. greater than confiscating her lands and removing
her French servants, Paul Doherty cites a document now held in The National
Archives in Kew: E 403/201, membranes 14-15. I've sent a request to the
National Archives that these membranes be digitised and emailed to me,
and I'll be most interested to see what they actually say, but even a
cursory look at the document E 403/201 on the National Archives website
reveals that it is part of the Issue Rolls of Roger de Waltham for the
sixteenth regnal year of Edward II, that is, 8 July 1322 to 7 July
1323. Roger de Waltham was the keeper of Edward's wardrobe from May
1322 to October 1323.  So whatever this document says about
Edward and Isabella's children and their households, it cannot relate
to Edward 'removing' them from Isabella's care in or about September 1324 as Doherty and other writers following him state, as 'cruelty' towards her or otherwise.
The date of the setting up of John of Eltham's household under the control of Eleanor Despenser cannot in fact be established precisely, or even vaguely. The only evidence we have that Eleanor was in charge of the household of the king and queen's second son comes from an undated roll of expenses now also held in the National Archives, E 101/382/12, which bears the title 'Expenses in the household of Eleanor Despenser (who had the care of John of Eltham).' As this roll is undated, clearly we cannot know when the care of John of Eltham or the control of his household was given to Eleanor, and it could have happened at any time between the boy's birth in August 1316 and near the end of his father's reign (in early October 1326, Edward II and Hugh Despenser the Younger fled London and left ten-year-old John in nominal charge of the Tower, with Eleanor really in charge). Lisa Benz St John has recently suggested that Eleanor's custody of John might only have occurred temporarily in the summer of 1320, when Edward II and Isabella both travelled to France for a month for Edward to pay homage to her brother Philip V in Amiens for his French lands, and may have left their younger son in Eleanor's care.  Nothing on this expense roll confirms the story that John of Eltham was deliberately removed from Isabella's care in the autumn of 1324 and given to Eleanor to punish the queen.
The undated roll of expenses might also belong to the period 1325/26, with two other household accounts of John of Eltham which survive in the National Archives and which are dated to Edward II's nineteenth regnal year, which ran from 8 July 1325 to 7 July 1326, and his twentieth regnal year, which began on 8 July 1326 and was brought to a premature end by the invasion of the queen and Roger Mortimer and Edward's forced abdication in January 1327.  Queen Isabella was in France for almost the whole of the period covered by these two accounts, from early March 1325 until she and Roger Mortimer returned to England with their invasion force in September 1326, and thus was unable to care for her son at this time anyway. Eleanor Despenser, born Eleanor de Clare in 1292, was Edward II's eldest niece and wife of his chamberlain and 'favourite' Hugh Despenser the Younger, whom she had married in 1306. As the eldest granddaughter of Edward I, a great noblewoman and heiress, the daughter and sister of earls of Gloucester, and mother of at least nine or ten children of her own by the end of Edward II's reign, as well as someone to whom Edward II was extremely close and obviously trusted greatly (see this post), Eleanor seems an entirely suitable person to have the care of the king and queen's son, who was her own first cousin, albeit twenty-four years her junior.
One undated roll of expenses in which Eleanor Despenser (temporarily?) had the care of John of Eltham seems a remarkably thin basis for declaring that John was forcibly removed from Isabella as a punitive action against the queen in 1324. The Lanercost and Flores Historiarum chronicles say that Edward II and Hugh Despenser the Younger appointed Eleanor, Hugh's wife, as a kind of guardian over Isabella in 1324, charged with spying on her, carrying her seal and monitoring her correspondence. Whether this story is true or not is hard to say; the Westminster chronicle Flores was viciously hostile to Edward II, and the writer of Lanercost, up near the Scottish border, was far from court and writing a couple of decades later, so is hardly a reliable source for what was happening there. Assuming they are correct, however, it has never been explained by writers who believe and repeat the story and also follow Paul Doherty's story precisely how Eleanor is meant to have guarded Isabella so that she had no privacy, yet at the same time looked after John of Eltham somewhere away from the queen so that she never saw her son, given the usual (and entirely unfounded) assumption that Edward never or rarely allowed Isabella to see her children. It was normal for royal and noble boys to be raised in another household from the age of seven or thereabouts. In May 1301 Edward I ordered that his eldest grandchild Gilbert de Clare, future earl of Gloucester (and Eleanor Despenser's brother), then just past his tenth birthday, be sent to live in the household of Queen Marguerite, Gilbert's step-grandmother.  Edward I's daughter Elizabeth also happily sent her daughter Eleanor, future countess of Ormond, to live with Queen Marguerite, and the girl was later raised at Amesbury Priory at Edward II's expense and in the company of her cousin Joan Gaveston and her aunt, Elizabeth and Edward's sister Mary the nun.  There are numerous other examples, and Isabella herself had young male wards living in her household, boys whose tenant-in-chief fathers had died but whose mothers were often still alive. I don't understand why so many modern writers assume that Isabella's son must have been forcibly removed from her or why the queen would have thought there was anything wrong or unusual in John of Eltham being raised in his own household under the command of the king's (and therefore also her) eldest niece. Perhaps even Isabella herself appointed Eleanor Despenser to look after her son John, when she went to France in 1320 with the king or when she went to France alone in March 1325, or at some other date. Eleanor was a lady of Isabella's household in 1311/12, a year when the queen's household accounts happen to survive (most of them don't) and almost certainly in other years as well; she was with the queen at Tynemouth in October 1322 when they came close to being captured by the Scots; and the two women were still close enough in February 1323 to write a virtually identical letter to the treasurer on behalf of Roger Mortimer's wife Joan, which strongly implies that they met and discussed the matter together.
The date of the establishment of a separate household for Edward II and Isabella's daughters
Eleanor of Woodstock (b. June 1318) and Joan of the Tower (b. July 1321) likewise cannot be stated with certainty. The two girls were, by 3 March 1325, living at Marlborough Castle in the care of Hugh Despenser the Younger's sister
Isabel Hastings and her third husband Ralph de Monthermer, who had
previously been married to Edward II's sister Joan of Acre and thus had a good claim to being the
girls' uncle. (Oddly, Paul Doherty, in his haste to condemn Edward II's 'even greater cruelty' to Isabella, doesn't seem to realise that the woman he calls "another court favourite, Isabella Hastings" was in fact Hugh Despenser's sister, which would strengthen his argument that Edward and Hugh acted to punish the queen.) Ralph de Monthermer died in early April 1325 and Isabel Hastings kept the custody of the king and queen's daughters until at least 19 February 1326.  Looking at the entry on the Close Roll relating to this, it is possible that custody of Eleanor and Joan was indeed given to Isabel Hastings in September 1324 as Doherty claims, though it is impossible to say for certain and it may have happened earlier - perhaps sometime during Edward II's sixteenth regnal year, July 1322 to July 1323, to which year the Issue Rolls of Roger de Waltham Doherty cites as a source for his story actually belong (see above). By 17 August 1326, Eleanor and Joan
had a mestresse or governess, as evidenced in Edward II's chamber account: Joan
Jermy, or 'Jonete Germye' as the account calls her, the sister
of Edward's sister-in-law Alice Hales, countess of Norfolk (who had
married his half-brother Thomas of Brotherton in about 1321).  Edward II sent letters to his daughters at Marlborough on 25 July 1326, and presumably had seen them there when he attended the wedding of Isabel Hastings' daughter Margaret to his household knight Sir Robert Wateville on 19 May that year. 
Edward II and Isabella's eldest child Edward of Windsor, the future Edward III, had been set up with his own household almost from birth in November 1312, and both parents visited him on occasion. In January and again in October 1319, the six-year-old heir to the throne was granted the Derbyshire manor of High Peak and other lands by his father to cover the expenses of his brother John of Eltham and sister Eleanor of Woodstock, aged two and a half and seven months respectively in early 1319, "the king wishing that the said John and Eleanor stay in the company of the said Edward and at his expenses, as they have now done for some time."  Evidently Isabella was perfectly satisfied with the living arrangements of her children, as the grant of October 1319 to Edward of Windsor includes the words "by the king and order of the queen." On 1 May 1320, Edward II granted Isabella the manor of High Peak, Derbyshire, previously held by their son Edward of Windsor to cover the expenses of his brother and sister, as noted above: "Grant, during pleasure, to queen Isabella of the castle and honour of High Peak...to hold in aid of the expenses of John, the king's son, and Eleanor his sister, the king's daughter." 
This is actually the only direct evidence we have that Edward and Isabella's children were ever part of Isabella's household and that she was financially responsible for them. Although it is likely that she did at some point, there is no direct
evidence that Isabella ever had the care of her younger daughter Joan,
whatever mawkishly melodramatic scenes of the little girl being torn
from her protesting mother's arms modern novelists might like to
imagine. The three children were looked after by nurses, who were each granted an income of thirty pounds a year for life in September 1327, presumably by Isabella herself: Matilda de Perie in the case of John of Eltham and Joan of the Tower, Joan du Bois in the case of Eleanor of Woodstock.  Again, the basis for accusing Edward II of deliberately ripping the three children from Isabella's care in 1324, more than four years after the grant of High Peak to the queen for the expenses of two of them, is astonishingly thin. To quote Mark Ormrod in his article about the household of Edward III's young children, "The general assumption is that...the domestic needs of the younger siblings of the royal heir were financed out of the households of the king, the queen and/or their older brother...It is also reasonably evident that the younger royal children were formally under the custody of the queen, and that, whatever financial arrangements may have been made for them, they often moved around with their mother's itinerant household or were placed temporarily in the care of an individual appointed by the queen at some favoured royal residence...".  This, the younger children being formally under the queen's custody, is of course not at all the same thing as the queen having primary full-time care of her children, and does not mean that Isabella would have expected to raise her children alone for many years. As noted, royal and noble boys left their mother's custody at a very young age, and girls often when they married. Isabella sent her daughter Joan to marry David Bruce, future king of Scotland, in 1328, when the girl was only seven.
The people to whom Edward II, or perhaps even Isabella, gave custody of their children sometime in the 1320s were: Edward's niece Eleanor Despenser; Eleanor's sister-in-law Isabel Hastings; Edward's former brother-in-law Ralph de Monthermer, whose four children Mary, Joan, Thomas and Edward were the king's nieces and nephews and his children's first cousins; his sister-in-law Alice Hales' sister Joan Jermy. All of these people, therefore, were members of Edward II's extended family, and he had known Ralph de Monthermer since the latter married Edward's widowed sister Joan of Acre in early 1297 when the future king was only twelve. Edward may have known the Despenser siblings Hugh and Isabel, who were some years his junior, since childhood (they were grandchildren of William Beauchamp, earl of Warwick). Many modern writers seem to think that Hugh Despenser the Younger was a cross between the Antichrist and a genocidal psychopath, but to her contemporaries his sister Isabel Hastings was not tainted by this association: Edward II's niece Elizabeth de Burgh, Eleanor Despenser's sister, left her two daughters Isabella Verdon (aged ten) and Elizabeth Damory (aged nine) in Isabel's care when she attended Edward's funeral in December 1327, despite the hatred and anger she may have had for the late Hugh Despenser, who had treated her appallingly.  This strongly suggests to me that Isabel Hastings was known to be a maternal, trustworthy type, and therefore an entirely suitable person to look after the king and queen's daughters. Queen Isabella did not act in any way while she was in power between 1327 and 1330 to suggest that she thought Isabel Hastings had injured her in any way or failed in her duty towards the queen's daughters. I'm completely failing to see here how Edward can be deemed to have acted inappropriately; he gave the care of his and Isabella's children to people he knew well and trusted, and who were of sufficiently noble birth and position.
Further thoughts that occur to me: if Isabella thought that Edward and his 'favourite' Hugh Despenser had cruelly taken her children away from her in 1324, why didn't she accuse Despenser of it at his trial in November 1326? She accused him of everything else: persuading the king to reduce her income, sending her to France 'against the dignity of her estate', coming between herself and her husband, leaving her in danger of her life at Tynemouth, and so on. Her children were not mentioned at all. Neither did Isabella claim at any other time that her husband and his favourite had deprived her of her children. If they were taken from her and she suffered as much as modern writers claim she did, why did she never mention it? Why did the pope never mention it? Why did her brother Charles IV of France, who complained vociferously to Edward II about the removal of Isabella's French servants, never mention it? Why did no single fourteenth-century chronicler, several of whom wrote indignantly about the reduction of Isabella's income, mention that Edward 'stole' or 'cruelly removed' her children from her? Why is there absolutely no source to suggest that anyone believed that Edward, in setting up separate households for his younger children, had done anything out of the ordinary at all? The story doesn't appear even in sources hostile to Edward II, such as the Flores Historiarum, or continental writers such as Jean Froissart who (decades later) thought that Isabella secretly fled from England in 1325 because Edward was persecuting her. Funny that, isn't it? If Edward II had really done something so outrageous, hurtful and harmful to his queen, why does not one single fourteenth-century source say so?
There's an obvious answer as to why no source says that Edward II did anything wrong or unusual in setting up households for his children, of course. It's because absolutely no-one at the time or for a very, very long time afterwards thought he had done anything wrong or unusual. The notion that he did is an invention of the late twentieth century, based mostly or entirely on, rather mysteriously, a document now in the National Archives which dates to at least fourteen months before September 1324, the date Paul Doherty and others claim Edward maliciously and cruelly removed Isabella's children from her. If Isabella had been unhappy with her husband and Hugh Despenser because she thought they had done any such thing, we would surely know about it. The establishment of separate households for her three younger children, whenever this happened, does not mean that Isabella never or only rarely saw her children again or was not allowed contact with them. Why on earth would anyone assume that it did, or that she was somehow being punished? It baffles me. It was Isabella herself who refused to return to England and Edward II in late 1325 and decided to remain in France with her eldest son; her separation from her three younger children from then until she saw them again in October 1326 was therefore entirely her own choice. Although it does seem that Edward was being unnecessarily spiteful when he confiscated Isabella's lands in September 1324 and that this was indeed intended as a punitive act against her (as Isabella herself realised and pointed out at Hugh Despenser the Younger's trial in November 1326), there is no reason, none, to assume that Edward had deprived her of her children at any point prior to her departure or ever intended to do so. It was Isabella who used their son Edward of Windsor as a weapon against his father in 1326, not Edward who used their other children as a weapon against his wife.
In the summer of 1340, Edward III set up a household for his and Queen Philippa's children Isabella, Joan, Lionel and John, then aged eight, six, twenty months and a mere four months, under the care of one Isabel de la Mote. Joan (who was fated to die of plague in 1348 on her way to marry Pedro the Cruel of Castile) had previously been in the care of the dowager countess of Pembroke, Marie de St Pol.  Funnily enough, I've never seen anyone claim that Edward III was cruelly depriving his queen of her children or punishing her by doing this. After Isabella and Roger Mortimer's downfall in October 1330 and perhaps even before (it's generally very difficult to ascertain the whereabouts of children), Isabella's daughter Eleanor of Woodstock lived in the household of her sister-in-law Queen Philippa, not in her mother's.  Isabella in 1331 was under temporary house arrest after the execution of her 'favourite' Roger Mortimer and was being attended by a physician, yet no-one reproaches Edward III for 'cruelly removing' Isabella's daughter from her at a time when she was ill and grieving. The young king did not allow his mother to accompany Eleanor to the Low Countries when she married the count of Guelders in 1332; does anyone ever accuse him of cruelty towards Isabella for this reason? Do they heck.
When writing this post, the words 'The Isabella Exception' kept popping into my head; it seems to me that things which were entirely normal for pretty well every other royal and noble woman of the Middle Ages are far too often nowadays assumed to have been cruel and unusual when they happened to Isabella of France. For all the many things which Edward II did wrongly or badly, there were still plenty of ways in which he acted entirely in accordance with what was expected of him as a medieval king, man, husband and father and with the conventions of his world, and it is most unfair to condemn him for these things when others are not. Whatever some modern writers may like to think, Isabella wasn't parachuted into the Middle Ages from the twentieth or twenty-first century with modern attitudes towards motherhood and sexual equality and the like. She understood perfectly well the norms of the early fourteenth-century royal society in which she was born and raised and lived her entire life. Is it too much hope that writers might remember that her cultural and familial norms were vastly different to ours?
1) P[aul] C. Doherty, 'Isabella, Queen of England 1296-1330', unpublished D.Phil. diss., University of Oxford, 1977, p. 103; id., Isabella and the Strange Death of Edward II (London, 2003), p. 80.
2) T.F. Tout, The Place of the Reign of Edward II in English History (Manchester, 1914), pp. 159-160, 355.
3) Lisa Benz St John, Three Medieval Queens: Queenship and the Crown in Fourteenth-Century England (New York, 2012), p. 111.
4) The National Archives, E 101/381/12 and E 101/382/3.
5) Calendar of Patent Rolls 1292-1301, pp. 592, 606.
6) Benz St John, Three Medieval Queens, pp. 109-110; J.S. Hamilton, Piers Gaveston, Earl of Cornwall 1307-1312: Politics and Patronage in the Reign of Edward II (Detroit, 1988), p. 101.
7) Calendar of Close Rolls 1323-1327, p. 260; Patent Rolls 1324-1327, pp. 88, 157, 243.
8) Society of Antiquaries of London MS 122.
10) Calendar of Fine Rolls 1307-1319, p. 389; Fine Rolls 1319-1327, p. 6; Patent Rolls 1317-1321, p. 336.
11) Patent Rolls 1317-1321, p. 453.
12) Patent Rolls 1327-1330, p. 163.
13) W. M[ark] Ormrod, 'The Royal Nursery: A Household for the Younger Children of Edward III', English Historical Review, 120 (2005) pp. 400-401.
14) Frances Underhill, For Her Good Estate: The Life of Elizabeth de Burgh (New York, 1999), pp.40-41.
15) Ormrod, 'Royal Nursery', pp. 400, 410.
16) Mark Ormrod, Edward III (2011), p. 125; Patent Rolls 1330-1334, p. 78.