Hello,
Looks like you don't have enough RAM.. how much RAM did you allocate?
Another options is the values for:
# sysctl -a|grep shm
kernel.shmall = 2097152
kernel.shmmax = 33554432
Are too low. If so, try raising them.
Note that Nginx is only needed if you intend to use On-the-fly repackaging of MP4 files to DASH, HDS, HLS, MSS, otherwise, it isn't strictly necessary.
This means that you currently have libtomcat6-java installed whereas red5 requires libtomcat7-java.
You can check why the version 6 was installed with:
# aptitude why libtomcat6-java
if nothing you need needs it, you can remove it.
Note that Red5 is only needed if you intend to do web cam recording from the KMC, otherwise, it is not needed.