Aim Small, old-growth forest fragments generally have more small-seeded plants than large patches, due to the disappearance of large seed dispersing vertebrates. This pattern may differ for secondary forest fragments where differential migration ability rather than persistence of seed dispersers may be driving plant community assembly. In this study, we investigated the effect of habitat fragmentation on seed dispersers and plant community structure in regenerating forests. Location The Thousand Island Lake, China. Taxon Plants, birds and mammals. Methods We compiled diversity and abundance data for birds and mammals on islands in the Thousand Island Lake, China. We also surveyed the secondary plant communities and measured seed dispersal traits. Results Community-weighted mean seed size of woody plants decreased with island size. This pattern was related to compositional difference of the dispersers. We found that mammal diversity and abundance was only weakly or not related to island size; whereas bird diversity and abundance increased strongly with island size. Density of bird-dispersed plants was significantly positively related with island size. Since birds tend to disperse smaller seeds than mammals, the trend in seed size may have been a consequence of the shift in relative abundance of the two disperser guilds. Main Conclusions Differential responses of seed dispersers to habitat fragmentation may lead to pervasive shifts in the plant community structure of regenerating forest fragments. Our study highlights the importance of keeping large continuous forests in order to retain mammals and their dispersal capabilities.