Well, after running simulations with more nodes, I've determined that having all processes write to the same stdout terminal is hopeless. It's impossible to debug when you have more than a 1-2 processes interleaving debug printouts. Over the weekend, I started working on modifications to the simulator to have each node open its own console window when it starts. This removes the problem of interleaved print statements and makes things much easier to debug. Although it sounds easy, it is pretty difficult to open a new process in its own window. The only way I could figure out how to do it was to fork the process, send an exec command to open a new window, and have the node run as a standalone process inside the new terminal. If anyone has a better method, please let me know.
Anyways, the change means that I need to restructure the simulator and separate the node executable from the simulator executable. It's actually not too bad, and I've already done a lot of it. Since I was making modifications on the sim again, I decided to remove a lot of the hacks and actually try to turn it into a halfway decent application.
I was using pipes and the 'select' command to mux the data previously which was okay for a while, but started to get messy. Using pipes and select means that you're constantly polling the read pipe descriptors, and setting up the 'select' command is pretty ugly. If you're not familiar with the 'select' command, using it means that you have a lot of FD_SET and FD_ISSET macros in your function. In my opinion, the better way to do it is using threads to handle each of the pipe reads since you can just let them block instead of using the 'select' to poll them. It seems more natural to do it this way since the pipe communications were meant to block and the code flow is much better this way. I already converted one of the pipes to a thread and it made a big difference in how clean the code looks. So I decided to make the simulator a multi-threaded, multi-process application. Fun fun fun...
At least if I'm ever starving on the streets, I can try and get a job as a Linux system programmer.