A weak foundation at N4 will hurt your N3 performance. Good luck with your JLPT journey— ganbatte kudasai !