As any of the available SQL Server services, Reporting Services will use all of the server memory if needed, in some cases when you have multiple products configured for the same server, you need to limit these values, we use WorkingSetMaximum and WorkingSetMinimum configuration values to achieve this task.
Where to set these values?
Find the .config file
We need to localize the RsReportServer.config configuration file in our server, this file is in the SQL Server installation folder under MSRS00.XXXX\Reporting Services\ReportServer where the 00 is your reporting services version and the XXXXX are your instance name.
NOTE: Before changing any parameter in this file, also take a backup of the file contents first.
Find the Service tag
Once you have located the file, proceed to find the <Service></Service> tags, and add the <WorkingSetMinimum> and <WorkingSetMaximum> as desired, values are specified in kilobytes, so for example, to set a value of 4 GB you must put 4000000.
You can see a sample code of a min memory of 2GB and a max memory of 4 GB
When the value for WorkingSetMaximum is reached, the report server does not accept new request, so please have this in mind before configuring this value.
Also note that a service restart is required for the values to take effect.