Tons Of Data
-
NeverJustHere wrote:
(preferably wired)
How come ? Just for reference, in this case at least two different humans will be viewing the same camera, and as many as 25 at one time.
If someone figures out the security system is dependent on WiFi (or buys the details from a disgruntled ex-employee), then they can jam frequencies or perform denial of service attacks to mess with the system. Wired would not have this problem.
-
How could an "enterprise" (e.g., a multi-story hotel or a shopping mall parking lot supervisor or whatever) save a whopping quantity of data ? The quantity I'm considering would not work with on-site storage; this quantity would have to be off-site. Off the top of my head, I'm thinking... - One solid week (168 hours) - Five hundred video cameras - 720p continual video monitoring - Typical video quality for that resolution - Color, infrared, nite-vision, whatever, as the mood strikes Bandwidth alone would be really huge. Just did the math and a bluetooth connection wouldn't keep up with the first screen scan (anybody, please correct any bad arithmetic). Is this even possible in the first place ? Looks like 557,383,680,000 bytes of just data; no protocol or formatting, from one camera. Is that about half a Terabyte ?
I've briefly worked on such a system. Some include motion sensors along with the camera, while others analyze the video data. As long as no motion is detected, they omit the data from long-term storage, or only store a brief sample (one frame per second/minute, etc.). That, along with conventional video compression, reduces the storage requirements from outlandish to merely huge.
Software Zen:
delete this; -
How could an "enterprise" (e.g., a multi-story hotel or a shopping mall parking lot supervisor or whatever) save a whopping quantity of data ? The quantity I'm considering would not work with on-site storage; this quantity would have to be off-site. Off the top of my head, I'm thinking... - One solid week (168 hours) - Five hundred video cameras - 720p continual video monitoring - Typical video quality for that resolution - Color, infrared, nite-vision, whatever, as the mood strikes Bandwidth alone would be really huge. Just did the math and a bluetooth connection wouldn't keep up with the first screen scan (anybody, please correct any bad arithmetic). Is this even possible in the first place ? Looks like 557,383,680,000 bytes of just data; no protocol or formatting, from one camera. Is that about half a Terabyte ?
You can also configure most cameras to use 'Motion-Detection' setting. In my experience, most cameras are installed in places where pictures very rarely change.
Behzad
-
How could an "enterprise" (e.g., a multi-story hotel or a shopping mall parking lot supervisor or whatever) save a whopping quantity of data ? The quantity I'm considering would not work with on-site storage; this quantity would have to be off-site. Off the top of my head, I'm thinking... - One solid week (168 hours) - Five hundred video cameras - 720p continual video monitoring - Typical video quality for that resolution - Color, infrared, nite-vision, whatever, as the mood strikes Bandwidth alone would be really huge. Just did the math and a bluetooth connection wouldn't keep up with the first screen scan (anybody, please correct any bad arithmetic). Is this even possible in the first place ? Looks like 557,383,680,000 bytes of just data; no protocol or formatting, from one camera. Is that about half a Terabyte ?
The system that I've had experience allows you to set how often it captures an image. For security purposes you don't need a full video, just a good resolution picture of a face or license plate. So we have ours set to capture an image once per second and hold onto it for 60 days. We have about 30 cameras feeding into this system and it only has 2 terabyte of space in the hard drives, which we've never filled up.
-
Quote:
The quantity I'm considering would not work with on-site storage; this quantity would have to be off-site.
Why? What is the use case for the data? If it is purely for security, and will only be reviewed after an incident, then majority of it will be written, archived for a week, and finally deleted. Why set up a system to move all that data around? Keep it as close to the camera as possible. I'd look at a node consisting of simple PC, redundant storage, recording from several cameras, connected to dedicated security (preferably wired) network and accessible from central control station. Easier to add more cameras too. When access to the video is required, simply copy relevant files based on incident time to a central location.
At some point this could almost be easier and cheaper if you just hired some human to watch the camera and take appropriate action if whatever happens happens...
-
How could an "enterprise" (e.g., a multi-story hotel or a shopping mall parking lot supervisor or whatever) save a whopping quantity of data ? The quantity I'm considering would not work with on-site storage; this quantity would have to be off-site. Off the top of my head, I'm thinking... - One solid week (168 hours) - Five hundred video cameras - 720p continual video monitoring - Typical video quality for that resolution - Color, infrared, nite-vision, whatever, as the mood strikes Bandwidth alone would be really huge. Just did the math and a bluetooth connection wouldn't keep up with the first screen scan (anybody, please correct any bad arithmetic). Is this even possible in the first place ? Looks like 557,383,680,000 bytes of just data; no protocol or formatting, from one camera. Is that about half a Terabyte ?
Save it on removable harddrives, then sneakernet it to a secure location. For that quantity of data, you probably have to go to eSATA, as there probably isn't much else that will be fast enough. There are (or at least, were) a few enclosure systems that supported USB 2, eSATA and had a custom bay so they plugged into the system like a jumbo floppy. The USB2 would likely be convenient to provide offline access without requiring special workstation setups. Rather then try to handle it like a scheduled backup task, I'd set the system up to continually mirror the data onto the removable drives, then once a week swap them out. That way, the data path probably won't be a bottleneck.
We can program with only 1's, but if all you've got are zeros, you've got nothing.
-
How could an "enterprise" (e.g., a multi-story hotel or a shopping mall parking lot supervisor or whatever) save a whopping quantity of data ? The quantity I'm considering would not work with on-site storage; this quantity would have to be off-site. Off the top of my head, I'm thinking... - One solid week (168 hours) - Five hundred video cameras - 720p continual video monitoring - Typical video quality for that resolution - Color, infrared, nite-vision, whatever, as the mood strikes Bandwidth alone would be really huge. Just did the math and a bluetooth connection wouldn't keep up with the first screen scan (anybody, please correct any bad arithmetic). Is this even possible in the first place ? Looks like 557,383,680,000 bytes of just data; no protocol or formatting, from one camera. Is that about half a Terabyte ?
I used to use a program called GOTCHA! that would only record changes. So if you filmed yourself raising your hand and holding there for ten minutes before lowering it, on playback you'd just see you raise your hand and immediately drop it. For interior monitoring it was great. Outside not as much because winds and shifting shadows would trigger it. But you could mask the screen to mark areas to ignore (like bushes) and areas to watch (like sidewalks) That could reduce the recordings quite a bit. Changed images are timestamped so you can see when motion occurred. I used it once to monitor a whiteboard that some chucklehead would alter my diagrams. I wanted to catch who was trying to sabotage my system designs. Instead, I caught a monthly cleaning crew having their way with my office and walking off with several hundred dollars of equipment.
Psychosis at 10 Film at 11 Those who do not remember the past, are doomed to repeat it. Those who do not remember the past, cannot build upon it.
-
How could an "enterprise" (e.g., a multi-story hotel or a shopping mall parking lot supervisor or whatever) save a whopping quantity of data ? The quantity I'm considering would not work with on-site storage; this quantity would have to be off-site. Off the top of my head, I'm thinking... - One solid week (168 hours) - Five hundred video cameras - 720p continual video monitoring - Typical video quality for that resolution - Color, infrared, nite-vision, whatever, as the mood strikes Bandwidth alone would be really huge. Just did the math and a bluetooth connection wouldn't keep up with the first screen scan (anybody, please correct any bad arithmetic). Is this even possible in the first place ? Looks like 557,383,680,000 bytes of just data; no protocol or formatting, from one camera. Is that about half a Terabyte ?
I have read a lot of the comments. Here are my 2 cents, having dealt with a lot of data, and some of the rules of imaging/cameras. 1) I definitely prefer hard wired cameras, WiFi can easily be attacked, and attacked locally. 2) Think bandwidth, you should have Cat 5 going into Gigabit or better switches 3) Certainly, every camera should be encrypting their output. 4) Because of hiccups with the OC3/External connection, you should have the ability to store some HOURS of video locally. Preferably 3-7 days if possible. 5) Consider only uploading extremely compressed feeds. 5 to 10 fps vs. 30fps. 6) Design for failure. When your RAID Degrades, it has to NOT become readonly! But transfer times will drop. Too many devices writing to one device will clog the pipe. We had to redesign a system from ONE server to 3 servers because the bandwidth of the incoming and outgoing network cards could never be maintained. 2 servers worked, but no redundancy, and once we had to redesign things to talk to different servers, we took the extra step. The project grew a bit, and the extra server helped. 7) Plan for growth. I don't care that the number of cameras as never changed. If you can't handle a few more cameras, you are too close to an edge. 10% more cameras would make me feel safe. A friend implemented a local storage of video, and external storage of a "snapshot" picture every 3 seconds. It was very very efficient. Some of the snapshots increased if there was motion. So, there are a lot of variables. The big question is always WHY? WHY do they want to keep the video? Do they need it for police, investigations, etc? Do they need to just prove somebody, or do they need to ID the person. The whys involved here should determine what approaches you can and should use. Many cameras can be lower quality, some will be required to pickup a license plate or other details, and again, using techniques that change the quality as activity is detected really help to "compress" the actual data. Tons of questions. Good luck.
-
I have read a lot of the comments. Here are my 2 cents, having dealt with a lot of data, and some of the rules of imaging/cameras. 1) I definitely prefer hard wired cameras, WiFi can easily be attacked, and attacked locally. 2) Think bandwidth, you should have Cat 5 going into Gigabit or better switches 3) Certainly, every camera should be encrypting their output. 4) Because of hiccups with the OC3/External connection, you should have the ability to store some HOURS of video locally. Preferably 3-7 days if possible. 5) Consider only uploading extremely compressed feeds. 5 to 10 fps vs. 30fps. 6) Design for failure. When your RAID Degrades, it has to NOT become readonly! But transfer times will drop. Too many devices writing to one device will clog the pipe. We had to redesign a system from ONE server to 3 servers because the bandwidth of the incoming and outgoing network cards could never be maintained. 2 servers worked, but no redundancy, and once we had to redesign things to talk to different servers, we took the extra step. The project grew a bit, and the extra server helped. 7) Plan for growth. I don't care that the number of cameras as never changed. If you can't handle a few more cameras, you are too close to an edge. 10% more cameras would make me feel safe. A friend implemented a local storage of video, and external storage of a "snapshot" picture every 3 seconds. It was very very efficient. Some of the snapshots increased if there was motion. So, there are a lot of variables. The big question is always WHY? WHY do they want to keep the video? Do they need it for police, investigations, etc? Do they need to just prove somebody, or do they need to ID the person. The whys involved here should determine what approaches you can and should use. Many cameras can be lower quality, some will be required to pickup a license plate or other details, and again, using techniques that change the quality as activity is detected really help to "compress" the actual data. Tons of questions. Good luck.
I have a professional home security camera system, 16 High Res Cameras. The 1 T disk can store for about 2 weeks. They store ONLY WHEN THEY SEE MOTION!@ Take a look at the off the shelf equipment at Security Camera Direct. Wire it!
The Irishman
-
I used to use a program called GOTCHA! that would only record changes. So if you filmed yourself raising your hand and holding there for ten minutes before lowering it, on playback you'd just see you raise your hand and immediately drop it. For interior monitoring it was great. Outside not as much because winds and shifting shadows would trigger it. But you could mask the screen to mark areas to ignore (like bushes) and areas to watch (like sidewalks) That could reduce the recordings quite a bit. Changed images are timestamped so you can see when motion occurred. I used it once to monitor a whiteboard that some chucklehead would alter my diagrams. I wanted to catch who was trying to sabotage my system designs. Instead, I caught a monthly cleaning crew having their way with my office and walking off with several hundred dollars of equipment.
Psychosis at 10 Film at 11 Those who do not remember the past, are doomed to repeat it. Those who do not remember the past, cannot build upon it.
This it ? http://www.gotchanow.com[^]
-
I have read a lot of the comments. Here are my 2 cents, having dealt with a lot of data, and some of the rules of imaging/cameras. 1) I definitely prefer hard wired cameras, WiFi can easily be attacked, and attacked locally. 2) Think bandwidth, you should have Cat 5 going into Gigabit or better switches 3) Certainly, every camera should be encrypting their output. 4) Because of hiccups with the OC3/External connection, you should have the ability to store some HOURS of video locally. Preferably 3-7 days if possible. 5) Consider only uploading extremely compressed feeds. 5 to 10 fps vs. 30fps. 6) Design for failure. When your RAID Degrades, it has to NOT become readonly! But transfer times will drop. Too many devices writing to one device will clog the pipe. We had to redesign a system from ONE server to 3 servers because the bandwidth of the incoming and outgoing network cards could never be maintained. 2 servers worked, but no redundancy, and once we had to redesign things to talk to different servers, we took the extra step. The project grew a bit, and the extra server helped. 7) Plan for growth. I don't care that the number of cameras as never changed. If you can't handle a few more cameras, you are too close to an edge. 10% more cameras would make me feel safe. A friend implemented a local storage of video, and external storage of a "snapshot" picture every 3 seconds. It was very very efficient. Some of the snapshots increased if there was motion. So, there are a lot of variables. The big question is always WHY? WHY do they want to keep the video? Do they need it for police, investigations, etc? Do they need to just prove somebody, or do they need to ID the person. The whys involved here should determine what approaches you can and should use. Many cameras can be lower quality, some will be required to pickup a license plate or other details, and again, using techniques that change the quality as activity is detected really help to "compress" the actual data. Tons of questions. Good luck.
Kirk 10389821 wrote:
We had to redesign a system from ONE server to 3 servers because the bandwidth of the incoming and outgoing network cards could never be maintained. 2 servers worked, but no redundancy, and once we had to redesign things to talk to different servers, we took the extra step.
I totally missed the concept you are trying to describe. More explanation is welcome.
Kirk 10389821 wrote:
- Plan for growth.
Smirk ! Mind reader ??? Kreskin ? If not, you and I are definitely parallel thinkers. That's a key pillar of the business concept. From what we're seeing in the logistics of software, hardware, and firmware, plus the connections (an internet connection is integral to this) that's also the single biggest problem to overcome. In fact, we're probably spending more time on this one problem (adding more cameras) than all the others, as we definitely want the customer to do exactly that.
Kirk 10389821 wrote:
The big question is always WHY?
Nice brain. We in the gang here will have a philosophical round table about those sorts of concepts you've described. Thanks for the input; on all of your post. I will upvote it.
-
I have a professional home security camera system, 16 High Res Cameras. The 1 T disk can store for about 2 weeks. They store ONLY WHEN THEY SEE MOTION!@ Take a look at the off the shelf equipment at Security Camera Direct. Wire it!
The Irishman
Are these the guys ? http://www.securitycamerasdirect.com[^]
-
Are these the guys ? http://www.securitycamerasdirect.com[^]
Yes I've been using their 16 camera, High res with IR for about 4 years There very good with any help Just be sure to get any adapters for the cable 12 volt power to the camera
The Irishman
-
Kirk 10389821 wrote:
We had to redesign a system from ONE server to 3 servers because the bandwidth of the incoming and outgoing network cards could never be maintained. 2 servers worked, but no redundancy, and once we had to redesign things to talk to different servers, we took the extra step.
I totally missed the concept you are trying to describe. More explanation is welcome.
Kirk 10389821 wrote:
- Plan for growth.
Smirk ! Mind reader ??? Kreskin ? If not, you and I are definitely parallel thinkers. That's a key pillar of the business concept. From what we're seeing in the logistics of software, hardware, and firmware, plus the connections (an internet connection is integral to this) that's also the single biggest problem to overcome. In fact, we're probably spending more time on this one problem (adding more cameras) than all the others, as we definitely want the customer to do exactly that.
Kirk 10389821 wrote:
The big question is always WHY?
Nice brain. We in the gang here will have a philosophical round table about those sorts of concepts you've described. Thanks for the input; on all of your post. I will upvote it.
C-P-User-3 wrote:
I totally missed the concept you are trying to describe. More explanation is welcome.
There was a life-lesson in here. The design had 100Mb incoming network card, and a 100Mb outgoing network card. Little did we know that the hard drive and the bus could not keep them busy. (This was circa 1998). We had incoming files coming through at full throttle. But when we turned on the outbound channel, it was maybe doing 20-25% of 100Mb. It took a lot of math and analysis to see we had the server not only reading but writing to the HD, and then reading other areas and trying to send it out via the network. Between the drive max throughput, and the BUS speed, we had over-estimated what we could accomplish. And testing it by testing only the incoming half and the outgoing half (separately) was not good enough to find the problem. (That seems obvious right now, and the life lesson is test it like it is ACTUALLY going to be used).
C-P-User-3 wrote:
nice brain. We in the gang here will have a philosophical round table about those sorts of concepts you've described. Thanks for the input; on all of your post.
Thanks... I hope it helps. For me, the WHY is the most important question. It is what allows you to solve problems in the most interesting ways. I often remind new clients that my job isn't to give them what they want, but to deliver what they need! That comes from getting into the why! As does knowing how to design the system properly in the first place. (Actually, we audited a compound that had video cameras with labels indicating the IP address of the camera. These were security cameras, at least they were supposed to be. I have no problem with labeling the equipment with Random TAG numbers that require a secure spreadsheet to look them up.)
-
How could an "enterprise" (e.g., a multi-story hotel or a shopping mall parking lot supervisor or whatever) save a whopping quantity of data ? The quantity I'm considering would not work with on-site storage; this quantity would have to be off-site. Off the top of my head, I'm thinking... - One solid week (168 hours) - Five hundred video cameras - 720p continual video monitoring - Typical video quality for that resolution - Color, infrared, nite-vision, whatever, as the mood strikes Bandwidth alone would be really huge. Just did the math and a bluetooth connection wouldn't keep up with the first screen scan (anybody, please correct any bad arithmetic). Is this even possible in the first place ? Looks like 557,383,680,000 bytes of just data; no protocol or formatting, from one camera. Is that about half a Terabyte ?
This is a little crazy. How about recording only when motion is detected? That alone would save tons of resources, at the expense of a little complexity.
-
This it ? http://www.gotchanow.com[^]
Yes. It's getting a little long in the tooth and the video format is non-standard, but it can get the job done.
Psychosis at 10 Film at 11 Those who do not remember the past, are doomed to repeat it. Those who do not remember the past, cannot build upon it.