How to solve the slow speed problem of Airtel USB modem?

Your question has three different queries in it. So let me try to answer each one of them.

Modem replacement USB dongle modem from any telecom operator is one off purchase. And they are not going to take it back. They may replace the modem if there are any issues with the device connectivity problems like USB being malfunctioned etc. However as per their contract terms they don't take the modem back only replace it. So your funds are not going to be returned. At the best you can make them replace the dongle under excuse of faulty USB port etc.

Speed The speed of 4G or 3G network is dependent on the area you're into. Not many towers around India are changed for the new LTE and 3G spectrum. So you're less likely to get reception for the data in consistent basis. In short your speed is going to be variable. Some of the places where 3G tower installation or the up gradation is complete may have better speed. Do note that due to expensive upgrades, telecom providers have yet to finish their 3G and LTE upgrades. There is no resolution on this point unless telecom companies act on this.

Making complaint In this case you have an option to either downgrade to 2G dongle and stick with unlimited plan for that. Or you can close the connection. Airtel network is not known for being responsive towards the customer complaints. Making complaints specific to the speed won't be well received. And at the most they are going to be ignored. You can close the connection but make sure you properly take care of connection closing issue. This should be done with very careful process because airtel may come back with new bills even after connection closure.

Well, the problem could either be with the modem itself, or it could also be due to the network issues. Either way, I do not think there is any way you can improve the speed on your own. Here are a few suggestions -

Contact customer care
Register a complaint with AirTel. Please note that AirTel is quite notorious for lack of response for speed related complaints. Use multiple channels to place your complaints. Complaining on their social media channels can have better feedback.

Check your network zone
Not all cell sites ( called towers in layman's language) in India have been upgraded to 3G or LTE. Check if your area has better coverage. If not, you are bound to have slow speed. It would be advisable to opt for another service provider with better coverage in your area. Check if you can unlock your AirTel USB modem. If it can, you use any other SIM with the modem. Please note that the steps involved may damage the modem. Expert advice is needed for the procedure.

Surrender
If everything fails, consider surrendering your AirTel connection. Please note that Air Tel has many harrowing stories of continued billing even after closing the connection. Follow the requisite procedures carefully to close the connection. Once you submit an application for the termination of the services, DO NOT use the modem - even for checking. AirTel has been known to coming up with claims that the service is under use.