No problem! Think of it like editing an image in photoshop. If you take a photo off the web that looks good and scale it up, it will look bad. If you scale it down it looks fine. Same applies for a 3D model since that’s exactly what you’re doing with the texture except in a 3D space. So if you have a character and his face is a small part of a 1024x1024 map and you show a closeup at HD 1080 resolution, you will see pixellation. This is why people sometimes use seperate maps for closeups or multiple maps, etc.
Also check that you’re dividing your model enough to have the same resolution as it had in Zbrush. So if you have a low res model in zbrush and 8 subdiv levels, you need to subdivide it 8 times with displacement map as well to get the same number of Polys back, or somewhere close and a normal map can do the trick.
Also consider using decimation if you can get away with it. It will put the detail into the areas that need it and you may not need to use displacement maps.