Had to print at a higher horizontal negative (-0.07) to fit the bearing in. It was the same type as the one suggested, but apparently the brand can matter too. All and all I printed three rolling segments to get the right fit.
I also had to shave quite a bit off the inside of the loop to get the bearing to roll without rubbing the edges, but in the end managed it. Getting the screw through the roller was also a hassle. Had to use a pair of pliers to keep the spacers stationary. Again, managed.
The bracket to the printer chassis was actually a pretty perfect fit though, so there's that. In the end it will work fine, just required a bit of extra effort. Part of this could easily be due to me using low end filament, though not all of it I think.