Game runs smoothly for me, not sure what you found clunky aside from some invisible walls, which is probably my only gripe with it atm. Movement and combat is one of the best aspects of the game for me.
As for the pvp, yeah, at level 4 you will have a very hard time, that is possibly why you were not running into very many lower levels as well. A lot of classes don't unlock their extra blocks and dodges until level 20 or higher so you will have no chance against someone that has unlocked everything at level 45. As far as I understand the matchmaking is based on both your rank and availability not level, so it's no surprise that you came up against max levels since a lot of people wait until then to have all their skills unlocked before jumping into 1v1 pvp.
Levels 1-20 are really still part of the learning phase, you barely even come across any hard fights etc. so you have enough time to learn and get use to your skills and combos, which become very important later on. For most classes you can't really maximize your damage unless you do your combos. There are no healer classes either so surviving difficult fights are largely based on how well you use your blocks, counters or dodges, depending on how your class is played.
I don't really see this as a problem though because leveling up if you follow the quests is really fast, speed levelers reach max level in less than a day and I managed to max out my second character over the course of a weekend without much trouble. You also unlock more movement abilities later on, so you can run up walls or dive straight down out of the sky while gliding, which is a lot of fun too.
So far I am loving the game and am planning to play it at launch as well. Aside form the art style which I really adore, the fast paced, almost arcade like combat is really addicting, not sure I am going to find another game that does this as well anytime soon.
I made a post asking about BnS before and got no responses so it seems the real issue is that there is simply not enough interest to do something guild wise.