NOTE: I am running – and this all mostly relates to – an older version of Zoneminder, the
Majority of the issues described here are thing of the past thanks to direct x264 passthrough recording that came with Zoneminder 1.32 and I highly recommend to upgrade if you are not using it for whatever reason yet.
So, little back story. I have a server (~~dual-core~~ quad-core i5 w/ 8 GB RAM) in my basement that has been running for 10+ years doing various tasks. Currently it’s my primary backup server, media server, file server, OpenVPN server and a ZoneMinder server. For almost a decade now, it has been running off of 6 pieces of 2 TB drives, that are partitioned 10 GB for system (RAID1) and the rest for data (RAID6). Everything runs within one Debian instance which I understand is not ideal. My next setup will be one big hypervisor with SSDs and ZoneMinder will get its own dedicated virtual machine. Until then, it is what it is.
On an unrelated note, over the last two years, bunch of stuff was stolen from my property. Apart from locking everything down from the second theft on I also had an idea to build a low-cost CCTV system (AliExpress IP cameras and whatever NVR platform I can find for Linux). I did a little bit of googling and basically when you want something Linux-based and open source, there is only one stable solution: ZoneMinder. Over the last year I also played with Shinobi (very early version that despite my then struggles with ZoneMinder I have passed on) and I briefly tried also a corporate grade software (can’t remember the name) that is free for up to 8 cameras that is supposedly great, but for that I would need dedicated Windows machine, which I don’t have and don’t want to set up.
Anyway, here are my observations that could help you get your ZoneMinder running smoothly. It took me a long time to get everything running as it should through trial and error (everything here seems so obvious now, but I probably took a lot of dead ends).
At first, I had three cameras and everything sitting on that RAID6 array, which was a pain. Even little things like someone starting a Plex session that started re-encoding a movie on that same array could have been enough for the whole ZoneMinder to plop dead because of the array utilization. Or even moving large files on and off server. But anyway, I could go on and on, but here are my tips. Order is random.
Monit daemon – so the service restarts if it turns off
Have a Monit daemon running that will restart ZoneMinder should it go down for any reason.
The Monit setting can be found e.g. here. You can even add a sendmail that will let you know ZoneMinder was restarted, and you can then investigate why. Huge help for this is Netdata that helps you quickly identify congestions within the system (by default it goes up to one hour into history with 1 second steps, but if you have enough memory you can go way deeper).
This was very useful before applying the further tips. I haven’t had the service go down in a few months now.
Don’t watch the live montage
Looking at live stream montage is fun, but try not to. Before I made another improvements even couple of streams were enough for ZoneMinder to start dropping frames.
What I found beneficial is a separate HTML page with a “manual” montage of live single frames, with a link to access individual full-fps live streams only when necessary.
Single frame can be accessed through (or just copy image address from Montage page and edit as necessary):
Alternatively you can create montage with 1-3 fps streams based on this:
Note that I have the
&scale= set to
100 (%), but you can scale it down to
25. I am just loading full resolution from the server and only then resizing it locally within the browser to roughly 33% because it looks way better on my Retina Macbook.
Don’t use motion detection
Initially, I ran ZoneMinder in the Modect setting. That means that event is only created and recorded when motion is detected in the video stream. I thought it will be great and the footage won’t take much space. However, that had few other problems:
- Shitty Aliexpress cameras with grainy night image were virtually impossible to fine tune anyway (might do a whole separate post on these)
- Even during the day when everything worked as it should (kind of), there were missing events in the video before and after the event that could be valuable
- It uses a lot of CPU and HDD
So I ended up deciding to just record everything (Record setting). It does take a lot of space, but should anything get stolen again, I can analyze even areas outside my property that otherwise would not have been in the motion detection area because it would be triggering false alarm every time someone walks by.
I plan on integrating physical motion detectors that would trigger event based on actual movement and not on image analysis. These can be built based on e.g. inexpensive ESP8266 board with cheap PIR sensor. I then may get a lot of cat-triggered events, but at least they will be more fun to watch that grain-triggered events where nothing happens.
Dedicated hard drive for the recorded footage
My initial setup where I was recording onto the RAID6 array was plain stupid, because it meant that all 6 disks would be spinning nonstop, congesting the array, slowing everything else is down, and/or the ZoneMinder stoppping when other heavy traffic was going on on the array.
Dedicated drive is the way to go. You can use either “special” NVR drives like WD Purple or Seagate SkyHawk series, but even basic 5400rpm WD Red works just fine. I am currently running WD Red 6 TB with average utilization under 30%, and that is recording from 9 cameras (1080p, fps ranging 5 to 8 depending on camera and the time of the day – the frame rate drops a little in IR mode). The total incoming stream from the cameras is only about 25 mbps.
How to set up dedicated drive properly is described here. Just note the process is different on versions 1.30.4 or older and 1.31 or newer! (I’ve had some issues where I did not notice they are different…)
Also, in Debian (and other Linux distros) there is 5% of space on every drive reserved for root. Turn that off on your ZoneMinder drive and tadah, you have 300 GB extra space (on 6 TB drive). Basically run
sudo tune2fs -m 0 /dev/sdX, where
/dev/sdX is of course your ZoneMinder drive. More info here.
Turn off SMART testing on the dedicated drive
This may not apply to everyone, but I personally have a smartmontools daemon (smartd) set up to perform short SMART self-test daily and long SMART self-test weekly and I have noticed a lot of dropped frames during the long self-test, which basically scans each sector on the disk. It runs with low priority, but when there is neverending stream of data that wants to be written down, it just doesn’t seem to work very well.
So I turned it off. I still get an email should any raw SMART value change and a lot of the disk’s surface is still overwritten in less than 5 days, so I am not overly concerned about that.
Of course, I am still running the risk that the disk will plop dead without any SMART warning, and since I knows how the world works, it will also happen the same night when someone steals all my mountain bikes. But, oh, well, I might do like raid1 or something in the future.
Anyway, to turn off the testing only on one specific disk in the system, open
/etc/smartd.conf and add in this line:
/dev/sdX -d ignore (of course don’t forget to change the
/dev/sdX accordingly). Save, and then restart smartmontools.
Offload the database to an SSD
I sort of did this for reasons unrelated to ZoneMinder per se, but after moving all my databases onto the cheapest SSD I could find (WDS120G2G0A) I noticed significantly better performance while doing anything in ZoneMinder (should it be running
zmaudit.pl, scrolling through recorded events or even watching the live montage).
Which is not surprising, after all, before the database ran off of a raid-1 array that was duplicated onto six slow 5400 rpm drives.
I since then moved a whole bunch of other stuff onto the SSD and it was one of the best unplanned upgrades that I ever did to the server (I use it also for stuff like Plex server and it’s temporary re-encoding folder). (Yeah yeah I know, today, everyone uses SSD as a system disk even in home servers today, but my little server runs since the pre-SSD era!).
Limit FPS on the cameras
Rule of thumb is that 5 fps is good enough for basic surveillance. If your camera allows it, limit the fps in camera to 5-7 fps from the default 15-20. I personally have the FPS (where possible) set to 7 fps, which during the night in IR mode drops to around five fps.
Do NOT limit the FPS in ZoneMinder settings, it will cause all kinds of weird artifacts. I’ve accidentally set it once for one camera and believed for several months the camera is faulty before I noticed the fps was limited in ZoneMinder and not in the camera.
Unfortunately, I have some AliExpress cameras that run 15 fps without any options to limit it down, so a way to make ZoneMinder to “drop every second frame” would be nice, but I suppose I will have to deal with differently; just haven’t figured out how exactly just yet. Ok, I’ve actually figured this one out. Even with camera where the seller told me FPS cannot be changed, it actually can be done. Only the setting has to be done in an Internet Explorer with ActiveX allowed.
And thats probably all the things that I could think of right now but I will edit this article along the way, but with these basic tweaks (basically: separated drive for events, database on SSD and using Record/Mocord instead of Modect) the ZoneMinder has been rock stable for the last few months. Edit: Running strong for over a year now.