Recently I was reading through the Polychronicon, a chronicle written around 1350 by Ranulph or Ralph Higden, a monk of Chester. It has this to say on the subject of Edward II:
"A handsome man, of outstanding strength...He forsook the company of lords, and fraternised with harlots, singers, actors, carters, ditchers, oarsmen, sailors, and others who practise the mechanical arts...He was prodigal in giving, bountiful and splendid in living, quick and unpredictable in speech...savage with members of his household, and passionately attached to one particular person, whom he cherished above all..." (Bold mine). 
The part 'savage with members of his household' immediately grabbed my attention. I've also seen it quoted as 'lashed out at members of his household' or as 'cruel to his household'. Frankly I find this quite astonishing. There's no doubt at all that Edward II had a vile temper, as did most of the Plantagenet kings, and certainly he was a capricious and unpredictable person prone to difficult moods - he can't have been an easy man to be around sometimes - but I've never seen any confirmation anywhere else that he was ever actually violent, and definitely not with members of his own household. All the evidence I've seen from Edward's household accounts indicates mutual affection between the king and the men who served him closely. The only possible indications that he was capable of violence are, to my mind, unreliable: statements by Roger Mortimer and later Adam Orleton, bishop of Hereford, that if Queen Isabella returned to her husband in 1326/27 her life would be in danger from him, and in the very pro-Lancastrian Brut chronicle there's a passage wherein Edward was informed after his deposition that people suspected him of wanting to strangle his wife and son Edward III to death. He responded "God knows, I thought it never, and now I would that I were dead! So would God that I were! For then were all my sorrow passed."  Edward's horrified reaction, that he would rather be dead than have people think him capable of murdering his wife and child, is surely an indication that he had never thought such a thing. Isabella needed an excuse in 1327 not to return to her lawful husband; claiming that he might potentially hurt or even kill her provided a cast-iron one, and Roger Mortimer and their ally Orleton are hardly unbiased witnesses.
I can't, however, conclusively prove that Higden was mistaken in his assertion. Perhaps Edward II did indeed lash out at members of his household when in a rage, and there's just no other evidence of it which survives. This lack of corroborating evidence in itself doesn't necessarily make Higden wrong, of course, and the rest of his description of the king seems very accurate. (Higden also says that Edward habitually drank too much, and spilled state secrets while in his cups. That strikes me as entirely plausible.) I'm inclined to think, however, that at least in this instance, Higden confused Edward with his father. We do know that Edward I assaulted servants on occasion - he had to pay twenty marks' compensation to a squire at his daughter Margaret's wedding in 1290 after hitting him with a stick - and then there are the famous stories when he assaulted his own son Edward of Caernarfon near the end of his life and pulled out handfuls of his hair, and the earlier occasion when he was so exasperated with his daughter Elizabeth he tore the coronet off her head and threw it in the fire. There's also Edward I's cruelty as a young man, when for example he and some of his followers had another young man they encountered mutilated for very little reason. Here, for the record, is an example of Edward II's temper: on one occasion in the 1320s he flew into such a screaming rage with his ally Walter Reynolds, archbishop of Canterbury, that the archbishop pretended that he had to make an urgent visitation to his cathedral in order to escape from the king's presence. OK, shouting and ranting in an archbishop's face isn't very pleasant behaviour, but given that Edward's great-great-grandfather Henry II had the archbishop of Canterbury assassinated and his great-grandson Henry IV had the archbishop of York executed, it's hardly that bad. Edward II's first cousin Sancho IV of Castile, incidentally and to put Edward's relations with his barons into some kind of perspective, killed dissident nobles with his own hands. (Which was perhaps something Edward wished was possible in England in the weeks and months after Piers Gaveston's execution.)
As for Edward being cruel or savage to his household in ways that didn't necessarily involve violence, well, perhaps, but I don't really see it. I've pored over Edward II's household accounts and only see the king's frequent generosity towards his servants. The fact that many members of his household joined the Dunheveds' attempts to free him from Berkeley in 1327 and the earl of Kent's plot to free him in 1330 - willing to help him long after his downfall and even years after his alleged death - doesn't indicate to me that he had been cruel towards them.
In conclusion, no, I tend not to think that Edward II was physically violent towards anyone and especially not his servants, despite his temper and occasional rages. If he had been, I'm sure we'd have more evidence of it in chronicles, or, as we do for his father, records of compensation paid to injured servants (even the king wasn't allowed to hit people with impunity!). There is, for example, a record of Edward of Caernarfon as prince of Wales in February 1303 paying four shillings in compensation to his Fool Robert Bussard or Buffard for accidentally injuring him by playing some unspecified trick on him while they were swimming in the river at Windsor. Although it seems highly likely that Edward drank too much sometimes, alcohol seems to have made him liable to talk too much rather than aggressive. For all Edward II's numerous faults, I rather doubt that being violent was one of them, and I think that probably Ranulph Higden mistakenly had his father in mind when he thought that Edward assaulted his servants.
1) Polychronicon Ranulphi Higden, monachi Cestrensis, ed. Joseph Rawson Lumby (1857), vol. viii, p. 299.
2) The Brut or the Chronicles of England, ed. F. W. D. Brie (1906) vol. 1, pp. 252-253.