Website And Service Using Real Time Data Feed
-
I am about to start developing a new website, and I am a little stuck as to the correct architecture to use to achieve what I need. I have been given access to a data feed, which provides me with live, real time data. I need to cache this data and do some processing on it, and I can only create one connection to this service - it will block any subsequent attempts from the same IP. Therefore directly connecting via a website is not going to be feasible. It looks like I will need to create two separate parts to this project - a back end service that retrieves the data and performs any processing required, then a separate website that accesses the processed data. As the data is real time - the ability to update the web page in real time would also be desirable. As well as this, I also anticipate adding a few mobile apps in future, which will also need to connect to the back end system. Does anyone have any suggestions on what technologies to use? Obviously Json would be ideal for the inter-process communications, but what kind of back end technologies can I use that easily supports json while still allowing a persistent service that can maintain a permanent connection to the data source? Anyway, i'm rambling a bit. I hope it's clear, and I look forward to people's advice.
-
I am about to start developing a new website, and I am a little stuck as to the correct architecture to use to achieve what I need. I have been given access to a data feed, which provides me with live, real time data. I need to cache this data and do some processing on it, and I can only create one connection to this service - it will block any subsequent attempts from the same IP. Therefore directly connecting via a website is not going to be feasible. It looks like I will need to create two separate parts to this project - a back end service that retrieves the data and performs any processing required, then a separate website that accesses the processed data. As the data is real time - the ability to update the web page in real time would also be desirable. As well as this, I also anticipate adding a few mobile apps in future, which will also need to connect to the back end system. Does anyone have any suggestions on what technologies to use? Obviously Json would be ideal for the inter-process communications, but what kind of back end technologies can I use that easily supports json while still allowing a persistent service that can maintain a permanent connection to the data source? Anyway, i'm rambling a bit. I hope it's clear, and I look forward to people's advice.
Member 849873 wrote:
the ability to update the web page in real time would also be desirable. As well as this, I also anticipate adding a few mobile apps in future, which will also need to connect to the back end system.
Depends on your definition of "real time" but humans don't operate in "real time" anyways. So for example it is pointless to attempt to update a GUI 5 times a second just because the data is received that quickly.
Member 849873 wrote:
can I use that easily supports json while still allowing a persistent service that can maintain a permanent connection to the data source?
You need requirements and an architecture design before deciding on "technologies". For example the following questions need to be answered - Exactly how fast does this data arrive? - Does it need to be retained (persisted)? If so for how long? - The user apps do what with the data? Graph it? Scroll a bunch of numbers? What? - How many users will there be? And then ask this again REALISTICALLY how many users will there be? - Based on the above information what are realistic long term volume needs? This includes storage and network. - Does the data feed stop/slow? Or specifically how do you detect if the connection has stopped receiving data? Additionally does the source allow you to remain connected for long periods of time? (It might require a reconnect or a heart beat message.) - How will admin occur? For example if the connection goes down does someone need to be notified? If so how? If the main site goes down does someone need to be notified? How will you know of the main site went down? (Obviously these could be answered by waiting for the users to complain but that might not be ideal.)
-
Member 849873 wrote:
the ability to update the web page in real time would also be desirable. As well as this, I also anticipate adding a few mobile apps in future, which will also need to connect to the back end system.
Depends on your definition of "real time" but humans don't operate in "real time" anyways. So for example it is pointless to attempt to update a GUI 5 times a second just because the data is received that quickly.
Member 849873 wrote:
can I use that easily supports json while still allowing a persistent service that can maintain a permanent connection to the data source?
You need requirements and an architecture design before deciding on "technologies". For example the following questions need to be answered - Exactly how fast does this data arrive? - Does it need to be retained (persisted)? If so for how long? - The user apps do what with the data? Graph it? Scroll a bunch of numbers? What? - How many users will there be? And then ask this again REALISTICALLY how many users will there be? - Based on the above information what are realistic long term volume needs? This includes storage and network. - Does the data feed stop/slow? Or specifically how do you detect if the connection has stopped receiving data? Additionally does the source allow you to remain connected for long periods of time? (It might require a reconnect or a heart beat message.) - How will admin occur? For example if the connection goes down does someone need to be notified? If so how? If the main site goes down does someone need to be notified? How will you know of the main site went down? (Obviously these could be answered by waiting for the users to complain but that might not be ideal.)
I was going to make the same comment in regards to the "real-time" aspect of things... it has a broad range of meanings depending on context. For DSP engineers, real-time is generally hardly feasible on a desktop (depending on rates), so there's hardly any point in trying to refresh anything a human will "see" and interpret in that context.
-
I am about to start developing a new website, and I am a little stuck as to the correct architecture to use to achieve what I need. I have been given access to a data feed, which provides me with live, real time data. I need to cache this data and do some processing on it, and I can only create one connection to this service - it will block any subsequent attempts from the same IP. Therefore directly connecting via a website is not going to be feasible. It looks like I will need to create two separate parts to this project - a back end service that retrieves the data and performs any processing required, then a separate website that accesses the processed data. As the data is real time - the ability to update the web page in real time would also be desirable. As well as this, I also anticipate adding a few mobile apps in future, which will also need to connect to the back end system. Does anyone have any suggestions on what technologies to use? Obviously Json would be ideal for the inter-process communications, but what kind of back end technologies can I use that easily supports json while still allowing a persistent service that can maintain a permanent connection to the data source? Anyway, i'm rambling a bit. I hope it's clear, and I look forward to people's advice.
Member 849873 wrote:
I have been given access to a data feed, which provides me with live, real time data. I need to cache this data and do some processing on it, and I can only create one connection to this service - it will block any subsequent attempts from the same IP. Therefore directly connecting via a website is not going to be feasible.
Let me rephrase that; you have a feed with data, and you can get the feed once. What does the word "realtime" do here?
Member 849873 wrote:
a back end service that retrieves the data
Once as you explained. Since it's a feed, there's no way to stay connected to that. Then what? Change your IP and fetch the feed with real-time data again?
Member 849873 wrote:
As the data is real time - the ability to update the web page in real time would also be desirable.
You mean that CHANGES made by an employer in record A should also change that employee-info "straight away", even while the manager is looking at the webpage?
Bastard Programmer from Hell :suss: if you can't read my code, try converting it here[^]
-
I am about to start developing a new website, and I am a little stuck as to the correct architecture to use to achieve what I need. I have been given access to a data feed, which provides me with live, real time data. I need to cache this data and do some processing on it, and I can only create one connection to this service - it will block any subsequent attempts from the same IP. Therefore directly connecting via a website is not going to be feasible. It looks like I will need to create two separate parts to this project - a back end service that retrieves the data and performs any processing required, then a separate website that accesses the processed data. As the data is real time - the ability to update the web page in real time would also be desirable. As well as this, I also anticipate adding a few mobile apps in future, which will also need to connect to the back end system. Does anyone have any suggestions on what technologies to use? Obviously Json would be ideal for the inter-process communications, but what kind of back end technologies can I use that easily supports json while still allowing a persistent service that can maintain a permanent connection to the data source? Anyway, i'm rambling a bit. I hope it's clear, and I look forward to people's advice.