When IIS 6.0 is running in worker process isolation mode, you can configure application pools recycling based on memory consumption. When the application pool has consumed the specified amount of memory, the worker process serving that application pool is recycled and the memory is freed.
Open IIS Manager
Select Application Pools and right click on the correct application pool for your Web site
Select Properties.
Select Recycling tab
To enable worker process recycling based on memory usage, set the following values based on your needs: Maximum Virtual Memory & Maximum Used Memory
Click OK