fix: Respect container memory and CPU limits from cgroups

[Mantisus]

Description

get_memory_info and get_cpu_info reported the host machine even when the crawler ran under a container limit, because /proc/meminfo is not namespaced. The autoscaler sized its budget from host RAM and kept scaling until the container got killed, and a container pinned to two cores read the load of the whole machine as idle. Both now come from the cgroup of the process, with nothing to configure.

Two decisions worth a look:

• The control files are resolved through /proc/self/mountinfo and /proc/self/cgroup instead of assuming /sys/fs/cgroup, which is what makes this work under --cgroupns=host.
• Several cgroups can limit the same process, as they do under Kubernetes. The tightest limit is the ceiling a budget has to fit under, and the utilization is taken from the level closest to running out.

Without a limit, and outside Linux, the host values are used as before. A crawler in a container whose limit is below the host RAM now gets a smaller max_memory_size, so its concurrency drops.

Issues

• Closes: Memory metrics ignore container memory limits (no cgroup awareness) #2095
• Closes: Crawlee increasing concurrency until it dies #1535

Testing

• Add new tests with a fake cgroup filesystem

[Copilot]

Pull request overview

This PR makes Crawlee’s CPU and memory metrics container-aware on Linux by reading cgroup v1/v2 limits/usage (instead of relying solely on host-wide /proc/meminfo / psutil), so the autoscaler sizes budgets and CPU utilization against the resources actually available to the running process.

Changes:

• Add a new cgroup discovery + metrics reader (src/crawlee/_utils/cgroup.py) that resolves controllers via /proc/self/mountinfo + /proc/self/cgroup and reads the tightest applicable limits.
• Update get_memory_info() / get_cpu_info() to prefer cgroup-scoped totals/usage when limits apply, falling back to host metrics otherwise.
• Add comprehensive unit tests with a fake cgroup filesystem, and document autoscaling behavior under resource limits.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
tests/unit/_utils/test_cgroup.py	Adds unit tests covering cgroup v1/v2 discovery and limit/usage semantics, plus system.py integration behavior.
src/crawlee/_utils/system.py	Switches CPU/memory “system-wide” metrics to cgroup-aware readings when applicable; keeps host fallbacks.
src/crawlee/_utils/cgroup.py	Implements cgroup controller discovery and reads memory/cpu limits + usage from the most relevant hierarchy levels.
docs/guides/scaling_crawlers.mdx	Documents how autoscaling respects cgroup CPU/memory limits when running under container/systemd constraints.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[vdusek]

If I am correct, all the tests run against a fake cgroup filesystem. Is there a way to cover this end-to-end? e.g., running a crawler in a container with --memory and/or --cpus set, and asserting get_memory_info() and get_cpu_info()? Or have you tested it manually? Also, we should make sure we cover both --cgroupns=private and --cgroupns=host.