[Experiment] 스레드 과다 생성 테스트(max user processes)

ulimit 명령어

• soft : 새로운 프로그램을 생성하면 기본으로 적용되는 한도

  ubuntu@ubuntu-server:~$ ulimit -a
  core file size          (blocks, -c) 0
  data seg size           (kbytes, -d) unlimited
  scheduling priority             (-e) 0
  file size               (blocks, -f) unlimited
  pending signals                 (-i) 15403
  max locked memory       (kbytes, -l) 65536
  max memory size         (kbytes, -m) unlimited
  open files                      (-n) 1024
  pipe size            (512 bytes, -p) 8
  POSIX message queues     (bytes, -q) 819200
  real-time priority              (-r) 0
  stack size              (kbytes, -s) 8192
  cpu time               (seconds, -t) unlimited
  max user processes              (-u) 15403
  virtual memory          (kbytes, -v) unlimited
  file locks                      (-x) unlimited
  

• hard : 소프트한도에서 최대로 늘릴 수 있는 한도

  ubuntu@ubuntu-server:~$ ulimit -aH
  core file size          (blocks, -c) unlimited
  data seg size           (kbytes, -d) unlimited
  scheduling priority             (-e) 0
  file size               (blocks, -f) unlimited
  pending signals                 (-i) 15403
  max locked memory       (kbytes, -l) 65536
  max memory size         (kbytes, -m) unlimited
  open files                      (-n) 1048576
  pipe size            (512 bytes, -p) 8
  POSIX message queues     (bytes, -q) 819200
  real-time priority              (-r) 0
  stack size              (kbytes, -s) unlimited
  cpu time               (seconds, -t) unlimited
  max user processes              (-u) 15403
  virtual memory          (kbytes, -v) unlimited
  file locks                      (-x) unlimited
  

현재 최대 생성 가능 스레드 개수(max user processes) 확인

ubuntu@ubuntu-server:~$ ulimit -a | grep processes
max user processes              (-u) 15403
ubuntu@ubuntu-server:~$ ulimit -aH | grep processes
max user processes              (-u) 15403

테스트를 위해 max user processes 낮추기

sudo vi /etc/security/limits.conf

*               soft    nproc           1024
*               hard    nproc           4096

재접속 후 max user processes 다시 확인

ubuntu@ubuntu-server:~$ ulimit -a | grep processes
max user processes              (-u) 1024
ubuntu@ubuntu-server:~$ ulimit -aH | grep processes
max user processes              (-u) 4096

api-example.jar 생성

@RestController
public class TestController {
    @GetMapping("/api/makeThreads/{threadCount}")
    public String makeThreads(@PathVariable("threadCount") int threadCount) {
        for (int i = 0; i < threadCount; i++) {
            int number = i;

            new Thread(() -> {
                System.out.println("start : " + number);

                try {
                    Thread.sleep(1000 * 60 * 20);
                } catch (InterruptedException e) {
                    e.printStackTrace();
                }

                System.out.println("finish : " + number);
            }).start();
        }
        return "OK";
    }
}

api-example.jar 실행

java -jar api-example.jar

api-example.jar에서 실행중인 스레드 개수 확인

ubuntu@ubuntu-server:~$ jps
3702 api-example.jar
4024 Jps

ubuntu@ubuntu-server:~$ ps -o thcount 3702
THCNT
   34

max user processes보다 많은 스레드 생성을 유발해보기

curl localhost:8080/api/makeThreads/2000

[292.497s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
2020-07-19 12:52:33.763 ERROR 3702 --- [io-18080-exec-1] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached] with root cause

java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
	at java.base/java.lang.Thread.start0(Native Method) ~[na:na]
	at java.base/java.lang.Thread.start(Thread.java:803) ~[na:na]
	at com.example.apiexample.TestController.makeThreads(TestController.java:39) ~[classes!/:0.0.1-SNAPSHOT]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
	at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]

결론

• 위와 같은 오류 발생시 ulimit 명령어로 max user processes가 너무 낮게 잡혀있지 않은지 확인
• 너무 낮게 잡혀있을 경우 limits.conf 파일의 nproc 값을 적절하게 조절하여 이슈 해결

참고

• https://woowabros.github.io/experience/2018/04/17/linux-maxuserprocess-openfiles.html