Every time you upload a distribution to PAUSE and it is mirrored to CPAN, the CPAN Testers download the distribution to test it. You can do this with an experimental version so PAUSE won’t index it and CPAN clients won’t know it is there. You can try something, see if it works, such as a patch for an operating system you don’t have, and when you are satisfied make it a real release.
To do this, simple put an underscore in your module version:
our $VERSION = '1.23_01';
The distribution won’t show up in the index files the clients see, but it will still be on CPAN.