Computers are complicated. Animation is complicated. 3D graphics are complicated. Using a computer to make 3D animation is just begging for trouble.
I come from hardworking salt-of-the-earth stock, optimized for simpler times and wilderness survival, and guided by a number of easily-memorized, frequently-rhyming dicta, including: “Never pay anybody to do anything you can do yourself for free.”
The trouble with that particular dictum is that it’s bunk. Doing things is never free — you pay in time, and thus in opportunity cost. It ought to be something more like “Never do anything you can pay somebody else to do while making more money than it costs to pay them,” but that’s not very snappy, and doesn’t dovetail quite as neatly into the Puritan work ethic.
My Space Bunnies animation was a labor of love, made on the cheap with second-hand machines and as much free software as possible. The choices I made — ostensibly to save myself time, money, and effort — ultimately cost me more of everything.
Horrible Farm Accident
My render farm was from the island of misfit toys. It was all shapes and sizes, heated my office on cold winter days, and sounded like a jet taking off. I chose this arrangement intentionally because outdated computers on Craigslist are cheap, pre-assembled, and generally have working operating systems pre-installed. I also understand that it’s better to have many smaller machines instead of a few big machines — even if the combined CPU power is the same — for reasons of overhead.
The theory goes that even if you have a single super-fast processor, there’s still a baseline amount of overhead in the form of cache-swapping, disk-writing, and ram-reading that must occur for each frame rendered, and it’s faster to distribute this overhead among multiple machines, even if total processing power of the farm remains the same.
This theory assumes a number of things about the hardware you use. By using Craigslist or Freegeek machines out of the back of a van, you’re letting other people make many hardware decisions for you, and with the wrong hardware, overhead may increase.
I attempted to replace the most egregiously slow drives, network cards, and ram in my farm, but I’m not a hardware specialist, and discrepancies in the farm’s performance suggest that I missed something somewhere. Most suspiciously, faster machines were sometimes taking more time to render jobs than slower machines. A few of them would render fine once they got going, but would take super-extra long to get started on a job. One of the machines would occasionally spontaneously shut down, even while idle.
I didn’t have the wherewithal to investigate and correct all the possible hardware problems, such as clock-speed incompatibilities between motherboards, processors, and memory, and as a consequence I’m left with the lingering suspicion that much of the farm was hamstrung throughout the project.
Unless you have expert hardware knowledge, it’s simpler and faster to buy pre-built machines, either from vendors like Dell or HP, or in pieces from Newegg, keeping to identical and guaranteed configurations. The savings in troubleshooting time will more than offset the extra cost, assuming you don’t squander it playing Halo 3.
A Tangled Web
My farm ran various flavors of Windows — mostly sundry versions of XP, whatever came pre-installed on the machines — and my primary workstation ran Win2K. I ought to have synchronized everything before I began, but I was in the middle of other work at the time and didn’t want to upgrade my primary machine midstream lest everything explode. I also didn’t expect the project to take two years, and by the time I realized there was a problem, I’d invested too much time to turn back. This melange of systems caused networking problems, as the varying versions of Windows handle permissions and paths in often-conflicting ways.
To distribute and manage the rendering of shots across the farm, I used a free, open-source render manager called SXSpider (Yahoo account required to view the SXSpider Yahoo group, which includes installer files).
I don’t recommend Spider. It’s really just a simple distributor, and doesn’t do much managing. It has complicated and unforgiving installation instructions which include walking in funny circles in a particular order, and if you miss a step you must start over from the beginning. This is apparently because of the way it bribes the Windows local networking and permissions structures to do its bidding. Highlights include the manual set-up of separate user accounts with specific names and permissions for each processor in a multi-processor system; it’s all quite kludgy.
Once installed, Spider was able to run a plurality of my jobs without mishap, though it aborted many others for reasons I was never able to fathom. If I was lucky, they’d fail randomly, in which case re-rendering fixed it. If I wasn’t lucky, the jobs would fail repeatedly, and I was forced to either spend a lot of time troubleshooting or just give in and render the shot manually from within Maya on my workstation. Either way, it took time I would otherwise have used to improve the piece, or possibly sleep. I was able to deduce that certain jobs started by computers running Windows 2000 wouldn’t run on machines running XP (or vice versa) or took orders of magnitude longer, because of the length of the names of the particle cache files. This is regressive, reactionary, and counter-revolutionary, and should be dragged off to the country to be re-educated.
Caches in general were a sticking point for Spider, mostly due to difficulties resolving path names across the network. And though it may not have been Spider’s fault, IK parent constraints never behaved as expected, whether cached and properly referenced or not. I was forced to restrict rendering of scenes with IK parent constraints to a single machine, all in one batch. And that single machine had to run the job all in one big batch, because if it took the frame range in smaller batches, the parent constraints would reset at the beginning of each one, and the first frame of each batch would be incorrect. Incorrect!
Spider fooled me by working every so often, but as with any abusive relationship, its underlying flaws always surfaced in a crisis, and I would have been better off paying for a real render manager. This ExtremeTech article liked the free DrQueue better than Spider, but DrQueue requires compiling. Compiling and manual setup are slow and difficult and require arcane knowledge, and I want fast and easy that I can do while dumb.
I used a utility called AMP Winoff to monitor my processor usage. Winoff allows CPU usage to trigger events such as shutdown or DOS commands. Combined with a command-line utility called sendemail, for particularly long renders I had my farm send a text message to my phone when it had finished.
After Effects was competent as a compositing application, though I’ve written previously about its limited ability to handle z-depth.
I spent a lot of time compensating for the poor render layer implementation of Maya 6. Mental Ray for Maya 6 won’t render anything that isn’t in the default render layer. Both of these issues were solved in Maya 7, and upgrading would have eliminated a lot of pain.
Don’t Go There
When considering this article, I first planned to detail the lengths I went to complete the project, the tricks I used to wrangle the Goldbergian spaghetti bomb I called my render farm, and the pitfalls and triumphs in my battle against the software, thus aiding the pilgrim who treads the same path — but as I went through my notes from the process, now many months in the past, I was amazed at the hoops I jumped through to get it done.
My advice to the amateur who wants to follow in my footsteps: Don’t. There’s a better way. I know the tools are expensive and you have no extra money, but if you know anything about computer graphics, your time is already worth too much to play this game.
If you really are in it for the love of the struggle, animation may be too rewarding for you; I suggest digging a well with your hands, or entering politics. If your goal is the production of animation, take the time you would spend trying to work with cheap software and hardware and make some more money instead. Perhaps you could work for an animation studio, honing skills and meeting people. From this perspective, even part-time work barking your knuckles for money is better in the long run than fruitlessly arguing with unfeeling machinery, full-time, for free.
With the money you earn, buy real tools. Hire people to do things for you. Investigate professional render farms (may I suggest ResPower), and learn to love paying for their services, which include customer support. While you’re at it, pay somebody to do your laundry. Buy a dishwasher. Take your car to a shop, or better yet, buy a car that requires less maintenance. In these ways you will accomplish more, finish sooner, and have more time to bask in the sweet, sweet glow of your own well-earned smugness.« previously: Inventors of the Doughnut | Home | next: MayaWikiOnline »