The guys at Volano have a very detailed report on this, but they don't provide a very good way to link to a summary of the relevant facts, so I'm going to provide them here
There are two key limits you may run into while running a Java server on Linux:
Believe it or not, the Linux kernel imposes a limit on the number of file descriptors a given process can have open at once. What's a file descriptor? Well, a file descriptor is a handle on an open file. It represents things like the mode the file was opened (such as read-only) and the current access position of the process in the file. While 1024 sounds like a lot of file descriptors (I mean how many files do you want to have going at the same time?), you need to realize that on Unix systems, lots of things appear as files (and therefore require file descriptors). These includes sockets, which Java uses for it's TCP/IP communications. Understand the problem now?
In /usr/include/linux/limits.h:
#define NR_OPEN 1024
should be
#define NR_OPEN 4096
In /usr/include/linux/posix_types.h:
#define __FD_SETSIZE 1024
should be
#define __FD_SETSIZE 4096
In /usr/include/bits/types.h:
#define __FD_SETSIZE 1024
should be
#define __FD_SET_SIZE 4096
Before these changes take affect you have to build a fresh kernel. Please see the Linux Kernel-HOWTO for details on how to do this. Keep in mind you may want to read up on the solution to the second part of this problem before building a fresh kernel.
You need to give processes the option of increasing their file descriptor limits:
In /etc/security/limits.conf add two lines:
* soft nofile 1024
* hard nofile 4096
In /etc/pam.d/login add:
session required /lib/security/pam_limits.so
Linux also imposes system-wide file descriptor limits. These can be changed at run time, but to automate the process, add the following lines to /etc/rc.d/rc.local. This will make the change automatically for you every time you boot:
echo 8192 > /proc/sys/fs/file-max
echo 24576 > /proc/sys/fs/inode-max
This problem only occurs if you are using native threads. Of course, if you are doing server development, you really want to have access to native threads because without them you are stuck doing I/O with one client at a time.
A Linux thread and a Linux process look a lot alike. Internally they are both tasks. Linux puts a limit on the maximum number of tasks on a system (512), and the maximum number of tasks for a given user on a system (256). This means you can't have more than 254 threads in a single program, and you can't have more than about 450 or so threads on an entire system. Since Java needs threads to do concurrent I/O operations, this can be a real problem.
Well, the easy solution is to move to the 2.4.x kernels, which don't have this problem.
Alternatively, you can modify your existing kernel's limits as follows:
In /usr/include/linux/tasks.h:
#define NR_TASKS 512
should be changed to
#define NR_TASKS 4090
#define MAX_TASKS_PER_USER (NR_TASKS/2)
should be changed to
#define MAX_TASKS_PER_USER NR_TASKS
Unfortunately, the Linux kernel isn't the only thing imposing this limit on you. The standard C library also imposes a limit of 1000 threads per process. So you have to modify it as well. So, go download your latest copy of Glibc sources and then follow these steps.
In linuxthreads/internals.h
#define STACK_SIZE (2 * 1024 * 1024)
should be changed to
#define STACK_SIZE (64 * PAGE_SIZE)
In linuxthreads/sysdeps/unix/sysv/linux/bits/local_lim.h
#define PTHREAD_THREADS_MAX 1024
should be changed to
#define PTHREAD_THREADS_MAX 8192
After this is completed you can begin the painfull (several hour long process) of rebuilding your glibc shared library. Have fun!