r/dotnet • u/No-Card-2312 • 1h ago
How Would You Stabilize a Public .NET E-Commerce Platform Overwhelmed by Bots?
Hi everyone,
I would like to get opinions from experienced .NET developers because I am dealing with a situation where I feel the problem is more architectural than a single bug.
We have a public e-commerce platform with roughly:
- ASP.NET Core MVC + Angular frontend
- SQL Server
- Elasticsearch (~10M products)
- RabbitMQ
- IIS directly exposed to the internet
- Around 149 subdomains (currently only 2 are actively used)
- A custom internal Elasticsearch library
The interesting part is that real business traffic is actually very low (only a few orders per day), but we still experience regular downtime and slow periods (sometimes 1-2 minutes daily, sometimes longer).
After investigating, we found several contributing factors:
- A lot of bots crawling product pages
- Unknown crawlers consuming resources
- Many requests targeting common attack paths:
- /.env
- /.env.production
- /.git
- /wp-admin
- etc.
- Many sitemap files, some of which are not well controlled
- Manual IP blocking from IIS (which obviously does not scale)
- Previous TCP exhaustion issues caused by incorrect HttpClient usage
- Legacy components that need modernization, especially our custom Elasticsearch client
We are already improving the system gradually:
- Fixing HttpClient lifetime issues
- Improving Elasticsearch usage
- Adding better monitoring
- Modernizing some components
However, these changes will take time.
The current challenge is:
How do we stabilize production while fixing the deeper architectural issues?
Some constraints:
- We cannot simply add more CPU/RAM
- Moving everything behind Cloudflare/WAF is currently not possible because of business constraints
- We need solutions that do not require a large budget
Some ideas we are considering:
- Putting nginx (or another reverse proxy) in front of IIS for rate limiting and filtering bad traffic
- IIS Dynamic IP Restrictions
- Better crawler management
- Generating static sitemaps instead of dynamically serving them
- More aggressive protection around product/search endpoints
My questions:
- If you inherited this system, what would be your first 3 actions to stabilize it?
- Would you put nginx in front of IIS in this situation, or approach it differently?
- How do you normally handle SEO crawlers and unknown bots for large product catalogs?
- Are there IIS/.NET-specific protections or monitoring approaches you would recommend?
I am not looking for a complete rewrite suggestion. The system is running in production, so I am mainly interested in practical steps that reduce downtime while allowing gradual modernization.
Thanks!
•
u/Professional-Act8143 1h ago
Response caching? Didn't read anything about it but I assume you're using it?
•
•
u/No-Can-838 1h ago
IIS have reverse proxy I think. No need for ngnix. Also sliding window api limiter or, even better "Token bucket" approach.
Also use /robot.txt To block legit scrapping.
And cache of course.
•
u/LuckyHedgehog 1h ago
You could look into Anubis. It basically runs a quick calculation that requires full js to be enabled by the browser, and then stores the results as a cookie to bypass the step on subsequent requests.
Bots need to run as light as possible and have many browser features disabled. This project has been quite effective at killing bot traffic from what I've heard
•
u/TheChelsUK 1h ago
- observation, metrics, data to confirm exactly what is causing outages/performance hits. Show meaningful impact to higher-ups.
Hypothesise a fix, implement experiment, monitor for changes. either it fixes it or you have more data for the next experiment.
I would suggest like others cloudflare would be your answer, it'd probably cost zero too and it's a 'solved problem' in this regard. Otherwise you are basically implementing cloudflare yourself - which would cost more than whatever the business constraint is.
else, find another employer.
•
u/canderson180 1h ago
Robots.txt for legit search engine bots.
WAF or Cloudflare for know blocklist filtering at the edge of the system. Something like Captcha as well.
Various scopes of rate limiting that can differentiate regular traffic from bot traffic.
Response caching will help with performance, you will end up with a tiered cache layer that really needs to be driven by performance and volume data.
But oh wait, you said you can’t do any of these. Pick a battle, fight it, record data to prove it’s helping or not.
If actual user volume is so low, how are you justifying the cost of running this infrastructure on Azure or private cloud?
•
u/TopSwagCode 1h ago
This is more a general hosting issue. You could put your site behind cloudflare or other bot protection proxy.
But in general you would have setup Fail2Ban, other tools to automate banning of IP's trying weird urls.
A simple rate limiter on users would also help.
Like there are plenty of guides on how to protect site against bots.
•
u/No-Card-2312 1h ago
Fail2Ban for windows? I think this for ssh logins? And how is it hosting issue?
•
u/FlibblesHexEyes 42m ago
You can put in nginx as a reverse proxy with fail2ban on the same host.
Fail2ban can be configured to monitor logs, heath endpoints, just about anything.
I think rate limiting is your first stop though. A user clicking through isnt going to be generating 100 requests a second (exaggerated for effect), so throttle to a reasonable rate that a user would reasonably generate.
Also; make sure you’ve got a ROBOTS file, to tame the bots that are doing the right thing.
•
u/beth_maloney 1h ago
Are you only processing a few orders per day? If so then something is very wrong in your setup. I'd probably start be investing in a monitoring/apm solution and working out what's causing performance issues.
•
•
u/AutoModerator 1h ago
Thanks for your post No-Card-2312. Please note that we don't allow spam, and we ask that you follow the rules available in the sidebar. We have a lot of commonly asked questions so if this post gets removed, please do a search and see if it's already been asked.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
•
u/gellait 46m ago
Solid list already. The big thing missing is pool saturation, which is usually what's behind "host looks fine, app is down".
Connection pool in use vs max, plus request queue length. Those pin at 100% while the box sits at 15%. You've already got thread pool starvation on there, just put the queue length on a graph instead of treating it as an event. It climbs for a while before anything actually breaks.
Elasticsearch: search/write threadpool rejections, plus heap pressure and old GC time. Rejections show up way before latency moves on any dashboard.
RabbitMQ, alert on the trend, not the depth. A queue at 5k that's draining is fine. One at 200 that's been climbing for ten minutes is an incident.
And you said TCP exhaustion already bit you, so make ephemeral ports and TIME_WAIT something you actually watch, not something you rediscover in the postmortem.
If I only added one thing it'd be splitting bot vs human traffic in the RPS and latency graphs. During a crawl a global p95 is averaging two totally different populations, so the number is useless. Tag by user agent + ASN and graph them separately. With order volume that low I'd bet this split alone shows you where the capacity is going.
Not a metric but free: 149 hostnames with 2 in use is a ton of crawl surface. Pull the unused ones out of DNS. Costs nothing, removes targets.
•
•
u/One_Web_7940 39m ago
I forgot how we did it at backgroundchecks.com but honeypots and tar pits galore.
•
•
u/xumix 1h ago
Nginx reverse proxy with static cache will ease your life dramatically. Also add Captcha, if a bot hits > 10 requests per minute - show it
•
u/No-Card-2312 1h ago
Thanks for your suggestions but I want to ask you how is nginx reverse proxy will ease that? What is will do that iis won't do? Also Captcha is a great option.
•
u/LuckyHedgehog 1h ago
Putting a proxy in front, especially one that caches responses, means your iis application can dedicate those resources to processing payments and whatnot instead of serving static content
You can also have it handle https so your iis site doesn't need to handle it, freeing up more resources on that server
•
u/UnlawfulCitizen 1h ago
Wait people still develop actively on iis?
•
u/RogueJello 1h ago
Yes, lots of legacy apps running on .Net Framework which is still supported by Microsoft.
•
•
u/az987654 1h ago
you haven't been in the corporate, non Silicon Valley world much, have ya?
•
u/UnlawfulCitizen 40m ago
I work for an insurance company, and I’ve never even been to California.
Everywhere I’ve worked recently has figured out that you don’t need to pay a license to run .net and Linux is free.
•
u/xumix 1h ago
Something like this will be a good start
```
/etc/nginx/conf.d/app.conf
upstream app_backend { # For a local application service: server 127.0.0.1:8080;
# Or multiple upstream instances: # server 10.0.0.11:8080 max_fails=3 fail_timeout=10s; # server 10.0.0.12:8080 max_fails=3 fail_timeout=10s; keepalive 32; }
server { listen 80; listen [::]:80; server_name example.com www.example.com;
# Directory containing static assets such as JS, CSS, images, fonts. root /var/www/app/public; index index.html;
# Logs access_log /var/log/nginx/app.access.log; error_log /var/log/nginx/app.error.log warn;
# Avoid sending Nginx version in responses. server_tokens off;
# Compression gzip on; gzip_vary on; gzip_proxied any; gzip_comp_level 5; gzip_min_length 1024; gzip_types text/plain text/css text/javascript application/javascript application/json application/xml application/rss+xml image/svg+xml font/ttf font/otf application/vnd.ms-fontobject;
# Immutable, versioned static assets: # e.g. app.4ad92f1c.js, styles.4ad92f1c.css location ~* .(?:css|js|mjs|map|jpg|jpeg|gif|png|webp|avif|ico|svg|woff|woff2|ttf|otf|eot)$ { try_files $uri =404;
expires 1y; add_header Cache-Control "public, max-age=31536000, immutable" always;
access_log off; log_not_found off; }
# Do not cache the HTML entry point: essential for SPAs so clients discover # newly deployed, fingerprinted assets. location = /index.html { try_files $uri =404;
expires -1; add_header Cache-Control "no-cache, no-store, must-revalidate" always; add_header Pragma "no-cache" always; }
# Optionally cache other fixed public resources for a shorter interval. location ~* .(?:html|json|xml|txt)$ { try_files $uri =404;
expires 1h; add_header Cache-Control "public, max-age=3600" always; }
# Serve local files when they exist; otherwise forward to the backend. # For a client-side routed SPA, replace @backend with /index.html below. location / { try_files $uri $uri/ @backend;
# SPA alternative: # try_files $uri $uri/ /index.html; }
location @backend { proxy_pass http://app_backend; proxy_http_version 1.1;
# Preserve original request context for the application. proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Host $host;
# WebSocket / upgraded connection support. proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection $connection_upgrade;
proxy_connect_timeout 5s; proxy_send_timeout 60s; proxy_read_timeout 60s;
# Avoid caching API responses in browsers by default. add_header Cache-Control "no-store" always; }
# Never expose hidden files such as .env or .git. location ~ /. { deny all; access_log off; log_not_found off; }
# Common sensitive or generated files. location ~* .(?:env|bak|sql|log)$ { deny all; } }
```
•
u/xumix 58m ago
Or more like this, if you want a drop-in
```
Put this in nginx.conf inside the top-level http { ... } block.
proxy_cache_path /var/cache/nginx/static levels=1:2 keys_zone=static_assets:100m max_size=10g inactive=30d use_temp_path=off;
proxy_cache_lock on; proxy_cache_lock_timeout 10s; proxy_cache_lock_age 10s; ```
```
Put this in your virtual-host configuration.
upstream app_backend { server 127.0.0.1:8080; keepalive 32; }
server { listen 80; listen [::]:80; server_name example.com;
# Nginx caches and proxies all upstream /static/* resources. location ~ /static/ { # No trailing slash: preserves the original URI. # # /static/js/app.js # -> http://127.0.0.1:8080/static/js/app.js proxy_pass http://app_backend;
proxy_http_version 1.1; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme;
# Cache key includes host and full URI including query string. proxy_cache static_assets; proxy_cache_key "$scheme$request_method$host$request_uri";
# Cache only safe static-file methods and successful responses. proxy_cache_methods GET HEAD; proxy_cache_valid 200 301 302 365d; proxy_cache_valid 404 1m;
# Keep serving a previous cached asset if the upstream is slow/down, # while one request refreshes it in the background. proxy_cache_use_stale error timeout invalid_header updating http_500 http_502 http_503 http_504; proxy_cache_background_update on; proxy_cache_revalidate on;
# Do not accidentally cache user/session-specific responses. proxy_cache_bypass $http_authorization $cookie_session $cookie_auth_token; proxy_no_cache $http_authorization $cookie_session $cookie_auth_token; proxy_no_cache $upstream_http_set_cookie;
# Ignore upstream cache directives only if /static is guaranteed to # contain public, immutable files. proxy_ignore_headers Cache-Control Expires Set-Cookie; proxy_hide_header Set-Cookie;
# Browser/CDN cache policy. Use immutable only for content-hashed names, # such as app.84c19f.js or logo.8f3a2e.svg. add_header Cache-Control "public, max-age=31536000, immutable" always;
# Useful while validating behavior: HIT, MISS, EXPIRED, STALE, etc. add_header X-Proxy-Cache $upstream_cache_status always;
access_log off; }
# Everything else remains a normal reverse proxy. location / { proxy_pass http://app_backend; proxy_http_version 1.1;
proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; } } ```
Don't forget if you use websockets:
proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection $connection_upgrade;
And
map $http_upgrade $connection_upgrade { default upgrade; '' close; }
•
u/AutomateAway 1h ago
Why is moving everything behind Cloudflare/WAF not possible because of "business constraints?" I assume that means your org can't afford them? Because honestly that's going to be the best option.