The most common problems listed by the 56% who reported issues included price hikes, poor speeds, connection dropouts and router issues.
Which? polled more than 1,900 customers across 12 providers for its latest broadband satisfaction survey.
Virgin Media customers were the most likely to have experienced a problem, with 73% reporting some kind of difficulty in the last year, typically due to price.
Some 62% of Sky customers reported a problem, followed by 61% of BT customers, Which? said.
Zen Internet achieved the lowest proportion of complaints, with 25% of customers saying they had experienced a problem.
The most common complaint was around recent price increases, affecting one in five (22%) of broadband customers.
Virgin Media customers were twice as likely to have had a problem with price hikes (47%), but it was also the top complaint in relation to BT, with 30% of its customers reporting an increase.
Slow speed was the second biggest problem, affecting 21% of those polled.
TalkTalk customers were more likely to experience slow speeds than any other provider included in the survey, with 31% affected.
Almost one in five people overall (17%) suffered connection dropouts and the same percentage experienced problems with their router.
Virgin Media customers were the most likely to be left with no internet at all for hours or even days at a time (15%), this was also a problem for one in 10 (11%) of Post Office customers.
Other issues included customers struggling to resolve queries with their provider (6%), problems getting in touch with their provider (5%), a long wait for an engineer to visit (3%), incorrect or unexpectedly high bills (3%) and exceeding the usage limit (2%).
BT, Sky, TalkTalk, Virgin Media and Zen Internet - who together serve around 90% of landline and broadband customers in the UK - have agreed to automatically compensate customers affected by slow repairs, missed appointments and delayed installations from early next year under Ofcom plans.