So, little back story. I have a server (~~dual-core~~ quad-core i5 w/ 8 GB RAM) in my basement that has been running for 10+ years doing various tasks. Currently it’s my primary backup server, media server, file server, OpenVPN server and a ZoneMinder server. For almost a decade now, it has been running off of 6 pieces of 2 TB drives, that are partitioned 10 GB for system (RAID1) and the rest for data (RAID6). Everything runs within one Debian instance which I understand is not ideal. My next setup will be one big hypervisor with SSDs and ZoneMinder will get its own dedicated virtual machine. Until then it is what it is.
On an unrelated note, over the last two years, bunch of stuff was stolen from my property. Apart from locking everything down from the second theft on I also had an idea to build a low-cost CCTV system (AliExpress IP cameras and whatever NVR platform I can find for Linux). I did a little bit of googling and basically when you want something Linux-based and open source, there is only one stable solution: ZoneMinder. Over the last year I also played with Shinobi (very early version that despite my then struggles with ZoneMinder I have passed on) and I briefly tried also a corporate grade software (can’t remember the name) that is free for up to 8 cameras that is supposedly great, but for that I would need dedicated Windows machine, which I don’t have.
Anyway, here are my observations that could help you get your ZoneMinder running smoothly. It took me a long time to get everything running as it should through trial and error (everything here seems so obvious now, but I probably took a lot of dead ends).
At first, I had three cameras and everything sitting on that RAID6 array, which was a pain. Even little things like someone starting a Plex session that started re-encoding a movie on that same array could have been enough for the whole ZoneMinder to plop dead because of the array utilization. Or even moving large files on and off server. But anyway, I could go on and on, but here are my tips. Order is random.
Have a Monit daemon running that will restart ZoneMinder should it go down for any reason.
The Monit setting can be found e.g. here. You can even add a sendmail that will let you know ZoneMinder was restarted, and you can then investigate why. Huge help for this is Netdata that helps you quickly identify congestions within the system (by default up to one hour into history by 1 second steps, but if you have enough memory you can go way deeper).
Don’t watch the live montage
Looking at live stream montage is fun, but try not to. Before I made another improvements even couple of streams were enough for ZoneMinder to start dropping frames.
What I found beneficial is a separate HTML page with a “manual” montage of live single frames, with a link to access individual full-fps live streams only when necessary.
Single frame can be accessed through (or just copy image address from Montage page and edit as necessary):
Alternatively you can create montage with 1-3 fps streams based on this:
Note that I have the
&scale= set to
100 (%), but you can scale it down to
25. I am just loading full resolution from the server and only then resizing it locally within the browser to roughly 33% because it looks way better on my Retina Macbook.
Decide whether you really need motion detection
Initially, I ran ZoneMinder in the Modect setting. That means that event is only created and recorded when motion is detected in the video stream. That had few problems:
- Shitty Aliexpress cameras with grainy night image are impossible to fine tune anyway (might do a whole separate post on these)
- Even during the day when everything worked as it should (kind of), there were missing data before and after the event that could be valuable
- It uses a lot of CPU and HDD
So I ended up deciding to just record everything (Record setting). It does take a shitload of space, but should anything get stolen again, I can analyze even areas outside my property that otherwise would not have been in the motion detection area because it would be triggering false alarm every time someone walks by.
I plan on integrating physical motion detectors that would trigger event based on actual movement and not on image analysis. These can be built based on e.g. inexpensive ESP8266 board with cheap PIR sensor. I then may get a lot of cat-triggered events, but at least they will be more fun to watch that grain-triggered events.
Dedicated hard drive for the recordings
My initial setup where I was recording onto the RAID6 array was plain stupid, because it meant that all 6 disks would be spinning nonstop, congesting the array, slowing everything else is down, and/or causing the ZoneMinder to stop when other traffic was going on on the array.
Dedicated drive is the way to go. You can use either “special” NVR drives like WD Purple or Seagate SkyHawk series, but even basic 5400rpm WD Red works just fine. I am currently running WD Red 6 TB with average utilization under 20%, and that is recording from 7 cameras (720p-1080p, fps ranging 5 to 8 depending on camera), the total incoming stream from the cameras is only 12-16 mbps.
How to set up dedicated drive properly is described here. Just note the process is different on versions 1.30.4 or older and 1.31 or newer! (I’ve had some issues with not noticing that.)
Also, in Debian (and other Linux distros) there is 5% of space on every drive reserved for root. Turn that off on your ZoneMinder drive and tadah, you have 300 GB extra space (on 6 TB drive). Basically run
sudo tune2fs -m 0 /dev/sdX, where
/dev/sdX is of course your ZoneMinder drive. More info here.
Turn off SMART testing on the dedicated drive
This may not apply to everyone, but I personally have a smartmontools daemon (smartd) set up to perform short SMART self-test daily and long SMART self-test weekly and I have noticed a lot of dropped frames during the long self-test, which basically scans the surface of the whole drive. It should run with low priority, but when there is neverending stream of data that wants to be written down, it just doesn’t seem to work very well.
So I turned it off. I still get an email should any raw SMART value change and a lot of the disk’s surface is still overwritten in less than 5 days, so I am not overly concerned about that.
To turn it off only for one specific disk in the system, open
/etc/smartd.conf and add in this line:
/dev/sdX -d ignore. Save, and then restart smartmontools.
Offload database to an SSD
I sort of did this for reasons unrelated to ZoneMinder per se, but after moving all my databases onto the cheapest SSD I could find (WDS120G2G0A) I noticed better performance while doing anything in ZoneMinder (should it be running
zmaudit.pl, scrolling through recorded events or even watching the live montage).
Which is not surprising, after all, before the database ran off of a raid-1 array that was duplicated onto six slow 5400 rpm drives.
And I since moved a whole bunch of other stuff onto the SSD and it was one of the best unplanned upgrades that I ever did to the server (I use it also for stuff like Plex server and it’s temporary re-encoding folder). (Yeah yeah I know, today everyone uses SSD as a system disk even in home servers, but my machine runs since the pre-SSD era!)
Limit FPS on the cameras
Rule of thumb is that 5 fps is good enough for basic surveillance. If your camera allows it, limit the fps in camera to 5-7 fps from the default 15-20. I personally have the FPS (where possible) set to 7 fps, which during the night in IR mode drops to around five fps.
Do not limit the FPS in ZoneMinder camera settings, it will cause all kinds of weird artifacts.
~~Unfortunately, I have some AliExpress cameras that run 15 fps without any options to limit it down, so a way to make ZoneMinder to “drop every second frame” would be nice, but I suppose I will have to deal with differently; just haven’t figured out how exactly just yet.~~ Ok, I’ve actually figured this one out. Even with camera where the seller told me FPS cannot be changed, it actually can be done. Only the setting has to be done in a Internet Explorer with ActiveX allowed.
And thats probably all the things that I could think of right now but I will edit this article along the way, but with these tweaks (basically: separated drive for events, database on SSD and using Record/Mocord instead of Modect) the ZoneMinder has been rock stable for the last few months. Edit: Running strong for over a year now.