Network communication in the raw
-
Something I take for granted is that my browser will connect to my servers which will connect to other servers in my hosting setup and that there will be a form of communication that will allow data to be requested and received. But I have no idea how it all works at a low level. Zero. I know that my code will make use of the .NET TCP/IP libraries and that they will handle the work of talking to the OS, which will invoke the drivers, which will use my WiFi hardware, which will talk to my ISP, which will route the communication to the server I requested. The .NET libraries will take care of handling the handshaking to establish a connection, and the sending and receiving the data itself. But I've always wondering how this works. At low level. As in: what does the OS say to Broadcom driver to make the antennae do stuff. What commands does the driver send to the firmware? What (and how?) does the firmware do to make the antenna emit an electromagnetic signal and how does the firmware handle all the electromagnetic radiation it picks up to enable it to listen to the right signal? Another thing that's always bugged me is the nature of the (necessarily) discontinuous communication. It's not a continuous sine wave with a continuous stream of data going to and from. The data is broken into packets and there is acknowledgement and error correction in the packets being transmitted. Which implies some kind of state management going on. Which then raises the question of how does something like SignalR work when there are 10,000 connections (and what is a "connection"?) to a server and the data travels back and forth on a connection that's long term, not short term like a HTTP request (small furrowed brow thinking about HTTP Keep-alive) None of this is important to a developer, and yet it kind of is important to a developer in the same way garbage collection, disk access patterns and thread scheduling. You don't need to know them, but knowing them gives you the knowledge to make informed decisions. Does anyone know of a write-up of this stuff anywhere? Failing that: would anyone be interested in taking a stab at an article that walks through the life cycle of a bit of data travelling from code through the libraries to the OS to the drivers to the firmware to the hardware, via the ISP, across the routers, to the other and and back? I think it would be fascinating.
cheers Chris Maunder
-
Something I take for granted is that my browser will connect to my servers which will connect to other servers in my hosting setup and that there will be a form of communication that will allow data to be requested and received. But I have no idea how it all works at a low level. Zero. I know that my code will make use of the .NET TCP/IP libraries and that they will handle the work of talking to the OS, which will invoke the drivers, which will use my WiFi hardware, which will talk to my ISP, which will route the communication to the server I requested. The .NET libraries will take care of handling the handshaking to establish a connection, and the sending and receiving the data itself. But I've always wondering how this works. At low level. As in: what does the OS say to Broadcom driver to make the antennae do stuff. What commands does the driver send to the firmware? What (and how?) does the firmware do to make the antenna emit an electromagnetic signal and how does the firmware handle all the electromagnetic radiation it picks up to enable it to listen to the right signal? Another thing that's always bugged me is the nature of the (necessarily) discontinuous communication. It's not a continuous sine wave with a continuous stream of data going to and from. The data is broken into packets and there is acknowledgement and error correction in the packets being transmitted. Which implies some kind of state management going on. Which then raises the question of how does something like SignalR work when there are 10,000 connections (and what is a "connection"?) to a server and the data travels back and forth on a connection that's long term, not short term like a HTTP request (small furrowed brow thinking about HTTP Keep-alive) None of this is important to a developer, and yet it kind of is important to a developer in the same way garbage collection, disk access patterns and thread scheduling. You don't need to know them, but knowing them gives you the knowledge to make informed decisions. Does anyone know of a write-up of this stuff anywhere? Failing that: would anyone be interested in taking a stab at an article that walks through the life cycle of a bit of data travelling from code through the libraries to the OS to the drivers to the firmware to the hardware, via the ISP, across the routers, to the other and and back? I think it would be fascinating.
cheers Chris Maunder
... then a miracle happens...
-
Something I take for granted is that my browser will connect to my servers which will connect to other servers in my hosting setup and that there will be a form of communication that will allow data to be requested and received. But I have no idea how it all works at a low level. Zero. I know that my code will make use of the .NET TCP/IP libraries and that they will handle the work of talking to the OS, which will invoke the drivers, which will use my WiFi hardware, which will talk to my ISP, which will route the communication to the server I requested. The .NET libraries will take care of handling the handshaking to establish a connection, and the sending and receiving the data itself. But I've always wondering how this works. At low level. As in: what does the OS say to Broadcom driver to make the antennae do stuff. What commands does the driver send to the firmware? What (and how?) does the firmware do to make the antenna emit an electromagnetic signal and how does the firmware handle all the electromagnetic radiation it picks up to enable it to listen to the right signal? Another thing that's always bugged me is the nature of the (necessarily) discontinuous communication. It's not a continuous sine wave with a continuous stream of data going to and from. The data is broken into packets and there is acknowledgement and error correction in the packets being transmitted. Which implies some kind of state management going on. Which then raises the question of how does something like SignalR work when there are 10,000 connections (and what is a "connection"?) to a server and the data travels back and forth on a connection that's long term, not short term like a HTTP request (small furrowed brow thinking about HTTP Keep-alive) None of this is important to a developer, and yet it kind of is important to a developer in the same way garbage collection, disk access patterns and thread scheduling. You don't need to know them, but knowing them gives you the knowledge to make informed decisions. Does anyone know of a write-up of this stuff anywhere? Failing that: would anyone be interested in taking a stab at an article that walks through the life cycle of a bit of data travelling from code through the libraries to the OS to the drivers to the firmware to the hardware, via the ISP, across the routers, to the other and and back? I think it would be fascinating.
cheers Chris Maunder
I call it "magic", and allow myself to be amazed, without asking too many questions.
".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
When you pry the gun from my cold dead hands, be careful - the barrel will be very hot. - JSOP, 2013 -
Something I take for granted is that my browser will connect to my servers which will connect to other servers in my hosting setup and that there will be a form of communication that will allow data to be requested and received. But I have no idea how it all works at a low level. Zero. I know that my code will make use of the .NET TCP/IP libraries and that they will handle the work of talking to the OS, which will invoke the drivers, which will use my WiFi hardware, which will talk to my ISP, which will route the communication to the server I requested. The .NET libraries will take care of handling the handshaking to establish a connection, and the sending and receiving the data itself. But I've always wondering how this works. At low level. As in: what does the OS say to Broadcom driver to make the antennae do stuff. What commands does the driver send to the firmware? What (and how?) does the firmware do to make the antenna emit an electromagnetic signal and how does the firmware handle all the electromagnetic radiation it picks up to enable it to listen to the right signal? Another thing that's always bugged me is the nature of the (necessarily) discontinuous communication. It's not a continuous sine wave with a continuous stream of data going to and from. The data is broken into packets and there is acknowledgement and error correction in the packets being transmitted. Which implies some kind of state management going on. Which then raises the question of how does something like SignalR work when there are 10,000 connections (and what is a "connection"?) to a server and the data travels back and forth on a connection that's long term, not short term like a HTTP request (small furrowed brow thinking about HTTP Keep-alive) None of this is important to a developer, and yet it kind of is important to a developer in the same way garbage collection, disk access patterns and thread scheduling. You don't need to know them, but knowing them gives you the knowledge to make informed decisions. Does anyone know of a write-up of this stuff anywhere? Failing that: would anyone be interested in taking a stab at an article that walks through the life cycle of a bit of data travelling from code through the libraries to the OS to the drivers to the firmware to the hardware, via the ISP, across the routers, to the other and and back? I think it would be fascinating.
cheers Chris Maunder
Chris Maunder wrote:
... I think it would be fascinating.
I see that all Australian traits have vanished. It's efectivbely a Public Holiday seeing as Monday is right before Christmas Day this year. I'd have thought you would be hooking in to the piss by now on a sunny Sunday afternoon there in downtown Toronto and not thinking anything work like until at least 02-01-2019.
Michael Martin Australia "I controlled my laughter and simple said "No,I am very busy,so I can't write any code for you". The moment they heard this all the smiling face turned into a sad looking face and one of them farted. So I had to leave the place as soon as possible." - Mr.Prakash One Fine Saturday. 24/04/2004
-
Something I take for granted is that my browser will connect to my servers which will connect to other servers in my hosting setup and that there will be a form of communication that will allow data to be requested and received. But I have no idea how it all works at a low level. Zero. I know that my code will make use of the .NET TCP/IP libraries and that they will handle the work of talking to the OS, which will invoke the drivers, which will use my WiFi hardware, which will talk to my ISP, which will route the communication to the server I requested. The .NET libraries will take care of handling the handshaking to establish a connection, and the sending and receiving the data itself. But I've always wondering how this works. At low level. As in: what does the OS say to Broadcom driver to make the antennae do stuff. What commands does the driver send to the firmware? What (and how?) does the firmware do to make the antenna emit an electromagnetic signal and how does the firmware handle all the electromagnetic radiation it picks up to enable it to listen to the right signal? Another thing that's always bugged me is the nature of the (necessarily) discontinuous communication. It's not a continuous sine wave with a continuous stream of data going to and from. The data is broken into packets and there is acknowledgement and error correction in the packets being transmitted. Which implies some kind of state management going on. Which then raises the question of how does something like SignalR work when there are 10,000 connections (and what is a "connection"?) to a server and the data travels back and forth on a connection that's long term, not short term like a HTTP request (small furrowed brow thinking about HTTP Keep-alive) None of this is important to a developer, and yet it kind of is important to a developer in the same way garbage collection, disk access patterns and thread scheduling. You don't need to know them, but knowing them gives you the knowledge to make informed decisions. Does anyone know of a write-up of this stuff anywhere? Failing that: would anyone be interested in taking a stab at an article that walks through the life cycle of a bit of data travelling from code through the libraries to the OS to the drivers to the firmware to the hardware, via the ISP, across the routers, to the other and and back? I think it would be fascinating.
cheers Chris Maunder
A book could be written on this topic. I don't think I want to write it though. I would start by reading up on the OSI model of network layers. The highest level is the application and it goes all the way down to signals on a wire or in radio waves. It can be very informative.
"They have a consciousness, they have a life, they have a soul! Damn you! Let the rabbits wear glasses! Save our brothers! Can I get an amen?"
-
Something I take for granted is that my browser will connect to my servers which will connect to other servers in my hosting setup and that there will be a form of communication that will allow data to be requested and received. But I have no idea how it all works at a low level. Zero. I know that my code will make use of the .NET TCP/IP libraries and that they will handle the work of talking to the OS, which will invoke the drivers, which will use my WiFi hardware, which will talk to my ISP, which will route the communication to the server I requested. The .NET libraries will take care of handling the handshaking to establish a connection, and the sending and receiving the data itself. But I've always wondering how this works. At low level. As in: what does the OS say to Broadcom driver to make the antennae do stuff. What commands does the driver send to the firmware? What (and how?) does the firmware do to make the antenna emit an electromagnetic signal and how does the firmware handle all the electromagnetic radiation it picks up to enable it to listen to the right signal? Another thing that's always bugged me is the nature of the (necessarily) discontinuous communication. It's not a continuous sine wave with a continuous stream of data going to and from. The data is broken into packets and there is acknowledgement and error correction in the packets being transmitted. Which implies some kind of state management going on. Which then raises the question of how does something like SignalR work when there are 10,000 connections (and what is a "connection"?) to a server and the data travels back and forth on a connection that's long term, not short term like a HTTP request (small furrowed brow thinking about HTTP Keep-alive) None of this is important to a developer, and yet it kind of is important to a developer in the same way garbage collection, disk access patterns and thread scheduling. You don't need to know them, but knowing them gives you the knowledge to make informed decisions. Does anyone know of a write-up of this stuff anywhere? Failing that: would anyone be interested in taking a stab at an article that walks through the life cycle of a bit of data travelling from code through the libraries to the OS to the drivers to the firmware to the hardware, via the ISP, across the routers, to the other and and back? I think it would be fascinating.
cheers Chris Maunder
Chris Maunder wrote:
would anyone be interested in taking a stab at an article
Rick above has the right idea. That's not an article; it's a book. We went over the OSI layers back when I was in college. While it might make for an interesting read all these years later, I actually don't want to start thinking about it at this level. Somehow I'm thinking I'd fall into the trap of knowing just enough to be dangerous, and then start trying to "optimize for the wrong things".
-
Something I take for granted is that my browser will connect to my servers which will connect to other servers in my hosting setup and that there will be a form of communication that will allow data to be requested and received. But I have no idea how it all works at a low level. Zero. I know that my code will make use of the .NET TCP/IP libraries and that they will handle the work of talking to the OS, which will invoke the drivers, which will use my WiFi hardware, which will talk to my ISP, which will route the communication to the server I requested. The .NET libraries will take care of handling the handshaking to establish a connection, and the sending and receiving the data itself. But I've always wondering how this works. At low level. As in: what does the OS say to Broadcom driver to make the antennae do stuff. What commands does the driver send to the firmware? What (and how?) does the firmware do to make the antenna emit an electromagnetic signal and how does the firmware handle all the electromagnetic radiation it picks up to enable it to listen to the right signal? Another thing that's always bugged me is the nature of the (necessarily) discontinuous communication. It's not a continuous sine wave with a continuous stream of data going to and from. The data is broken into packets and there is acknowledgement and error correction in the packets being transmitted. Which implies some kind of state management going on. Which then raises the question of how does something like SignalR work when there are 10,000 connections (and what is a "connection"?) to a server and the data travels back and forth on a connection that's long term, not short term like a HTTP request (small furrowed brow thinking about HTTP Keep-alive) None of this is important to a developer, and yet it kind of is important to a developer in the same way garbage collection, disk access patterns and thread scheduling. You don't need to know them, but knowing them gives you the knowledge to make informed decisions. Does anyone know of a write-up of this stuff anywhere? Failing that: would anyone be interested in taking a stab at an article that walks through the life cycle of a bit of data travelling from code through the libraries to the OS to the drivers to the firmware to the hardware, via the ISP, across the routers, to the other and and back? I think it would be fascinating.
cheers Chris Maunder
-
Something I take for granted is that my browser will connect to my servers which will connect to other servers in my hosting setup and that there will be a form of communication that will allow data to be requested and received. But I have no idea how it all works at a low level. Zero. I know that my code will make use of the .NET TCP/IP libraries and that they will handle the work of talking to the OS, which will invoke the drivers, which will use my WiFi hardware, which will talk to my ISP, which will route the communication to the server I requested. The .NET libraries will take care of handling the handshaking to establish a connection, and the sending and receiving the data itself. But I've always wondering how this works. At low level. As in: what does the OS say to Broadcom driver to make the antennae do stuff. What commands does the driver send to the firmware? What (and how?) does the firmware do to make the antenna emit an electromagnetic signal and how does the firmware handle all the electromagnetic radiation it picks up to enable it to listen to the right signal? Another thing that's always bugged me is the nature of the (necessarily) discontinuous communication. It's not a continuous sine wave with a continuous stream of data going to and from. The data is broken into packets and there is acknowledgement and error correction in the packets being transmitted. Which implies some kind of state management going on. Which then raises the question of how does something like SignalR work when there are 10,000 connections (and what is a "connection"?) to a server and the data travels back and forth on a connection that's long term, not short term like a HTTP request (small furrowed brow thinking about HTTP Keep-alive) None of this is important to a developer, and yet it kind of is important to a developer in the same way garbage collection, disk access patterns and thread scheduling. You don't need to know them, but knowing them gives you the knowledge to make informed decisions. Does anyone know of a write-up of this stuff anywhere? Failing that: would anyone be interested in taking a stab at an article that walks through the life cycle of a bit of data travelling from code through the libraries to the OS to the drivers to the firmware to the hardware, via the ISP, across the routers, to the other and and back? I think it would be fascinating.
cheers Chris Maunder
Easy stuff. The foot bone is connected to the ankle bone, etc, etc.
If you can keep your head while those about you are losing theirs, perhaps you don't understand the situation.
-
Chris Maunder wrote:
... I think it would be fascinating.
I see that all Australian traits have vanished. It's efectivbely a Public Holiday seeing as Monday is right before Christmas Day this year. I'd have thought you would be hooking in to the piss by now on a sunny Sunday afternoon there in downtown Toronto and not thinking anything work like until at least 02-01-2019.
Michael Martin Australia "I controlled my laughter and simple said "No,I am very busy,so I can't write any code for you". The moment they heard this all the smiling face turned into a sad looking face and one of them farted. So I had to leave the place as soon as possible." - Mr.Prakash One Fine Saturday. 24/04/2004
Why do you think I'm here thinking about radio waves? This is not a sober preoccupation. At least it shouldn't be.
cheers Chris Maunder
-
A book could be written on this topic. I don't think I want to write it though. I would start by reading up on the OSI model of network layers. The highest level is the application and it goes all the way down to signals on a wire or in radio waves. It can be very informative.
"They have a consciousness, they have a life, they have a soul! Damn you! Let the rabbits wear glasses! Save our brothers! Can I get an amen?"
I'd probably need the abridged version. Possibly a version involving sock puppets to explain the hard bits.
cheers Chris Maunder
-
Something I take for granted is that my browser will connect to my servers which will connect to other servers in my hosting setup and that there will be a form of communication that will allow data to be requested and received. But I have no idea how it all works at a low level. Zero. I know that my code will make use of the .NET TCP/IP libraries and that they will handle the work of talking to the OS, which will invoke the drivers, which will use my WiFi hardware, which will talk to my ISP, which will route the communication to the server I requested. The .NET libraries will take care of handling the handshaking to establish a connection, and the sending and receiving the data itself. But I've always wondering how this works. At low level. As in: what does the OS say to Broadcom driver to make the antennae do stuff. What commands does the driver send to the firmware? What (and how?) does the firmware do to make the antenna emit an electromagnetic signal and how does the firmware handle all the electromagnetic radiation it picks up to enable it to listen to the right signal? Another thing that's always bugged me is the nature of the (necessarily) discontinuous communication. It's not a continuous sine wave with a continuous stream of data going to and from. The data is broken into packets and there is acknowledgement and error correction in the packets being transmitted. Which implies some kind of state management going on. Which then raises the question of how does something like SignalR work when there are 10,000 connections (and what is a "connection"?) to a server and the data travels back and forth on a connection that's long term, not short term like a HTTP request (small furrowed brow thinking about HTTP Keep-alive) None of this is important to a developer, and yet it kind of is important to a developer in the same way garbage collection, disk access patterns and thread scheduling. You don't need to know them, but knowing them gives you the knowledge to make informed decisions. Does anyone know of a write-up of this stuff anywhere? Failing that: would anyone be interested in taking a stab at an article that walks through the life cycle of a bit of data travelling from code through the libraries to the OS to the drivers to the firmware to the hardware, via the ISP, across the routers, to the other and and back? I think it would be fascinating.
cheers Chris Maunder
Chris Maunder wrote:
None of this is important to a developer, and yet it kind of is important to a developer in the same way garbage collection, disk access patterns and thread scheduling. You don't need to know them, but knowing them gives you the knowledge to make informed decisions.
Actually I'm not so sure that is true, if it really matters (say real real-time apps) then there will be mechanisms developed to work with & around it. Consider database design, was a time the way hard disks worked mattered to bot the database server developers and the app developers, so people optimised with the hardware in mind. Now databases can be distributed, on different media, even on unknown media (cloud). Even the way new hardware works, for instance if you read up on SSD's vs disks even though the o/s treats them the same the old rules don't apply, yeah you may still play around witch caching/buffering but optimising say refresh block sizes/transfer speeds for the hardware, not a factor that matters Network low level magic: knowing how it works, well there's one common way it does it for lan, about 3 common different methods for wireless, about 5 different common methods for WAN... there's nothing to optimise because one network session may use any one or more of these at the same time. Knowing how your network connection at low level is working right now will not give any clue as to what will be happening later, and "later" could be hours, minutes or even milliseconds. At high level app level nothing about the current connection at lowest level is useful to the app because it may be different before it can even apply whatever optimisations or rules to cater for what it found. The speed is about the only useful factor, but even that doesn't dictate low level say packet sizes or negotiation/error correction overheads... yawn, got called in to monitor some app, waste of half a day that I could have used for sleeping, but whatever, charging for a full day even though it's only 10 AM and I'm done :)
-
Something I take for granted is that my browser will connect to my servers which will connect to other servers in my hosting setup and that there will be a form of communication that will allow data to be requested and received. But I have no idea how it all works at a low level. Zero. I know that my code will make use of the .NET TCP/IP libraries and that they will handle the work of talking to the OS, which will invoke the drivers, which will use my WiFi hardware, which will talk to my ISP, which will route the communication to the server I requested. The .NET libraries will take care of handling the handshaking to establish a connection, and the sending and receiving the data itself. But I've always wondering how this works. At low level. As in: what does the OS say to Broadcom driver to make the antennae do stuff. What commands does the driver send to the firmware? What (and how?) does the firmware do to make the antenna emit an electromagnetic signal and how does the firmware handle all the electromagnetic radiation it picks up to enable it to listen to the right signal? Another thing that's always bugged me is the nature of the (necessarily) discontinuous communication. It's not a continuous sine wave with a continuous stream of data going to and from. The data is broken into packets and there is acknowledgement and error correction in the packets being transmitted. Which implies some kind of state management going on. Which then raises the question of how does something like SignalR work when there are 10,000 connections (and what is a "connection"?) to a server and the data travels back and forth on a connection that's long term, not short term like a HTTP request (small furrowed brow thinking about HTTP Keep-alive) None of this is important to a developer, and yet it kind of is important to a developer in the same way garbage collection, disk access patterns and thread scheduling. You don't need to know them, but knowing them gives you the knowledge to make informed decisions. Does anyone know of a write-up of this stuff anywhere? Failing that: would anyone be interested in taking a stab at an article that walks through the life cycle of a bit of data travelling from code through the libraries to the OS to the drivers to the firmware to the hardware, via the ISP, across the routers, to the other and and back? I think it would be fascinating.
cheers Chris Maunder
Actually, this is the very reason I got a Computer Engineering degree instead of a Computer Science degree. I had enough experience with the software side of things on my own as a hobbyist I wanted to understand how the hardware level worked. I can remember in one class early on we couldn't get our circuit to work just right. Turns out it was because of the wires we were using were too long and so the signals weren't getting to the right places at the right times. It IS amazing how much stuff we can do with a bunch of changes in voltage.
Everyone is born right handed. Only the strongest overcome it. Fight for left-handed rights and hand equality.
-
I'd probably need the abridged version. Possibly a version involving sock puppets to explain the hard bits.
cheers Chris Maunder
Chris Maunder wrote:
Possibly a version involving sock puppets to explain the hard bits.
Would you prefer Agro or Elmo as the Compere?
Michael Martin Australia "I controlled my laughter and simple said "No,I am very busy,so I can't write any code for you". The moment they heard this all the smiling face turned into a sad looking face and one of them farted. So I had to leave the place as soon as possible." - Mr.Prakash One Fine Saturday. 24/04/2004
-
Chris Maunder wrote:
None of this is important to a developer, and yet it kind of is important to a developer in the same way garbage collection, disk access patterns and thread scheduling. You don't need to know them, but knowing them gives you the knowledge to make informed decisions.
Actually I'm not so sure that is true, if it really matters (say real real-time apps) then there will be mechanisms developed to work with & around it. Consider database design, was a time the way hard disks worked mattered to bot the database server developers and the app developers, so people optimised with the hardware in mind. Now databases can be distributed, on different media, even on unknown media (cloud). Even the way new hardware works, for instance if you read up on SSD's vs disks even though the o/s treats them the same the old rules don't apply, yeah you may still play around witch caching/buffering but optimising say refresh block sizes/transfer speeds for the hardware, not a factor that matters Network low level magic: knowing how it works, well there's one common way it does it for lan, about 3 common different methods for wireless, about 5 different common methods for WAN... there's nothing to optimise because one network session may use any one or more of these at the same time. Knowing how your network connection at low level is working right now will not give any clue as to what will be happening later, and "later" could be hours, minutes or even milliseconds. At high level app level nothing about the current connection at lowest level is useful to the app because it may be different before it can even apply whatever optimisations or rules to cater for what it found. The speed is about the only useful factor, but even that doesn't dictate low level say packet sizes or negotiation/error correction overheads... yawn, got called in to monitor some app, waste of half a day that I could have used for sleeping, but whatever, charging for a full day even though it's only 10 AM and I'm done :)
Very valid point(s). It helps to know something about subnets and addresses when you organize a network but that is a rather high level. I have found that over the years the only lower level networking issue I have had to explicitly deal with at any level was the limited size of a single Ethernet packet. Once I wrote a function to continue reading until all the data was received that became a non-issue. I still have to deal with it every time I come across another "protocol" but having that base available makes it very easy. The interesting thing is I don't deal with that size exactly, only whether I have all the data yet as it arrives in sections. I just have to figure out how to determine the amount of data to receive.
"They have a consciousness, they have a life, they have a soul! Damn you! Let the rabbits wear glasses! Save our brothers! Can I get an amen?"
-
Something I take for granted is that my browser will connect to my servers which will connect to other servers in my hosting setup and that there will be a form of communication that will allow data to be requested and received. But I have no idea how it all works at a low level. Zero. I know that my code will make use of the .NET TCP/IP libraries and that they will handle the work of talking to the OS, which will invoke the drivers, which will use my WiFi hardware, which will talk to my ISP, which will route the communication to the server I requested. The .NET libraries will take care of handling the handshaking to establish a connection, and the sending and receiving the data itself. But I've always wondering how this works. At low level. As in: what does the OS say to Broadcom driver to make the antennae do stuff. What commands does the driver send to the firmware? What (and how?) does the firmware do to make the antenna emit an electromagnetic signal and how does the firmware handle all the electromagnetic radiation it picks up to enable it to listen to the right signal? Another thing that's always bugged me is the nature of the (necessarily) discontinuous communication. It's not a continuous sine wave with a continuous stream of data going to and from. The data is broken into packets and there is acknowledgement and error correction in the packets being transmitted. Which implies some kind of state management going on. Which then raises the question of how does something like SignalR work when there are 10,000 connections (and what is a "connection"?) to a server and the data travels back and forth on a connection that's long term, not short term like a HTTP request (small furrowed brow thinking about HTTP Keep-alive) None of this is important to a developer, and yet it kind of is important to a developer in the same way garbage collection, disk access patterns and thread scheduling. You don't need to know them, but knowing them gives you the knowledge to make informed decisions. Does anyone know of a write-up of this stuff anywhere? Failing that: would anyone be interested in taking a stab at an article that walks through the life cycle of a bit of data travelling from code through the libraries to the OS to the drivers to the firmware to the hardware, via the ISP, across the routers, to the other and and back? I think it would be fascinating.
cheers Chris Maunder
You know what they say: knowledge is dangerous! ;P
Chris Maunder wrote:
Which implies some kind of state management going on.
What I found fascinating when simulating TCP/IP over satellite and dealing with rain fade (packet loss due to rain) is that the hardware (or more precisely, the software in the hardware) handles all the protocols for requesting retransmittal of packets. Something we take for granted is that packets will be received in the correct order, which is achieved by the hardware layer, packet numbering, and ACK's. By default, TCP/IP over satellite is darn slow, as every ~1400 byte packet requires an ACK, which is a 500ms round trip (250ms from ground to satellite to ground, 250ms back for the ACK.) We were simulating buffering packets on the ground and simulating the ACK so we could get a continuous stream of packets, and using a separate "Error ACK" to resend only the garbled packets. It worked quite well in the simulations, achieving near hardwired transmittal rates. That was the closest I got to diving into the bowels of how things talk to each other.
Latest Article - A Concise Overview of Threads Learning to code with python is like learning to swim with those little arm floaties. It gives you undeserved confidence and will eventually drown you. - DangerBunny Artificial intelligence is the only remedy for natural stupidity. - CDP1802
-
Something I take for granted is that my browser will connect to my servers which will connect to other servers in my hosting setup and that there will be a form of communication that will allow data to be requested and received. But I have no idea how it all works at a low level. Zero. I know that my code will make use of the .NET TCP/IP libraries and that they will handle the work of talking to the OS, which will invoke the drivers, which will use my WiFi hardware, which will talk to my ISP, which will route the communication to the server I requested. The .NET libraries will take care of handling the handshaking to establish a connection, and the sending and receiving the data itself. But I've always wondering how this works. At low level. As in: what does the OS say to Broadcom driver to make the antennae do stuff. What commands does the driver send to the firmware? What (and how?) does the firmware do to make the antenna emit an electromagnetic signal and how does the firmware handle all the electromagnetic radiation it picks up to enable it to listen to the right signal? Another thing that's always bugged me is the nature of the (necessarily) discontinuous communication. It's not a continuous sine wave with a continuous stream of data going to and from. The data is broken into packets and there is acknowledgement and error correction in the packets being transmitted. Which implies some kind of state management going on. Which then raises the question of how does something like SignalR work when there are 10,000 connections (and what is a "connection"?) to a server and the data travels back and forth on a connection that's long term, not short term like a HTTP request (small furrowed brow thinking about HTTP Keep-alive) None of this is important to a developer, and yet it kind of is important to a developer in the same way garbage collection, disk access patterns and thread scheduling. You don't need to know them, but knowing them gives you the knowledge to make informed decisions. Does anyone know of a write-up of this stuff anywhere? Failing that: would anyone be interested in taking a stab at an article that walks through the life cycle of a bit of data travelling from code through the libraries to the OS to the drivers to the firmware to the hardware, via the ISP, across the routers, to the other and and back? I think it would be fascinating.
cheers Chris Maunder
Computer Networks 5th (International Economy Edition) Paperback – January 9, 2010 by Andrew S. Tanenbaum (Author), David J. Wetherall (Author) Internetworking with TCP/IP Vol.1: Principles, Protocols, and Architecture - Douglas E. Comer Internetworking with TCP/IP Vol.2: Internals and Implementation Internetworking With TCP/IP Vol.3: Client-Server Programming And Applications Versions
Caveat Emptor. "Progress doesn't come from early risers – progress is made by lazy men looking for easier ways to do things." Lazarus Long
-
Chris Maunder wrote:
Possibly a version involving sock puppets to explain the hard bits.
Would you prefer Agro or Elmo as the Compere?
Michael Martin Australia "I controlled my laughter and simple said "No,I am very busy,so I can't write any code for you". The moment they heard this all the smiling face turned into a sad looking face and one of them farted. So I had to leave the place as soon as possible." - Mr.Prakash One Fine Saturday. 24/04/2004
Animal.
cheers Chris Maunder
-
Something I take for granted is that my browser will connect to my servers which will connect to other servers in my hosting setup and that there will be a form of communication that will allow data to be requested and received. But I have no idea how it all works at a low level. Zero. I know that my code will make use of the .NET TCP/IP libraries and that they will handle the work of talking to the OS, which will invoke the drivers, which will use my WiFi hardware, which will talk to my ISP, which will route the communication to the server I requested. The .NET libraries will take care of handling the handshaking to establish a connection, and the sending and receiving the data itself. But I've always wondering how this works. At low level. As in: what does the OS say to Broadcom driver to make the antennae do stuff. What commands does the driver send to the firmware? What (and how?) does the firmware do to make the antenna emit an electromagnetic signal and how does the firmware handle all the electromagnetic radiation it picks up to enable it to listen to the right signal? Another thing that's always bugged me is the nature of the (necessarily) discontinuous communication. It's not a continuous sine wave with a continuous stream of data going to and from. The data is broken into packets and there is acknowledgement and error correction in the packets being transmitted. Which implies some kind of state management going on. Which then raises the question of how does something like SignalR work when there are 10,000 connections (and what is a "connection"?) to a server and the data travels back and forth on a connection that's long term, not short term like a HTTP request (small furrowed brow thinking about HTTP Keep-alive) None of this is important to a developer, and yet it kind of is important to a developer in the same way garbage collection, disk access patterns and thread scheduling. You don't need to know them, but knowing them gives you the knowledge to make informed decisions. Does anyone know of a write-up of this stuff anywhere? Failing that: would anyone be interested in taking a stab at an article that walks through the life cycle of a bit of data travelling from code through the libraries to the OS to the drivers to the firmware to the hardware, via the ISP, across the routers, to the other and and back? I think it would be fascinating.
cheers Chris Maunder
Chris Maunder wrote:
would anyone be interested in taking a stab at an article that walks through the life cycle of a bit of data
The trouble is that it's one of the things that are very simple to use because of the almost ridiculously complicated and convoluted processes going on in the background. And where to stop.
I wanna be a eunuchs developer! Pass me a bread knife!
-
Actually, this is the very reason I got a Computer Engineering degree instead of a Computer Science degree. I had enough experience with the software side of things on my own as a hobbyist I wanted to understand how the hardware level worked. I can remember in one class early on we couldn't get our circuit to work just right. Turns out it was because of the wires we were using were too long and so the signals weren't getting to the right places at the right times. It IS amazing how much stuff we can do with a bunch of changes in voltage.
Everyone is born right handed. Only the strongest overcome it. Fight for left-handed rights and hand equality.
ZurdoDev wrote:
It IS amazing how much stuff we can do with a bunch of changes in voltage.
True enough. Voltage and polarity; that's all there is to it, really. (And that's essentially a complete computer manual in under 25 words.) (And it's about as useful as some of the cr@ppy manuals I've had to wade through.)
I wanna be a eunuchs developer! Pass me a bread knife!
-
You know what they say: knowledge is dangerous! ;P
Chris Maunder wrote:
Which implies some kind of state management going on.
What I found fascinating when simulating TCP/IP over satellite and dealing with rain fade (packet loss due to rain) is that the hardware (or more precisely, the software in the hardware) handles all the protocols for requesting retransmittal of packets. Something we take for granted is that packets will be received in the correct order, which is achieved by the hardware layer, packet numbering, and ACK's. By default, TCP/IP over satellite is darn slow, as every ~1400 byte packet requires an ACK, which is a 500ms round trip (250ms from ground to satellite to ground, 250ms back for the ACK.) We were simulating buffering packets on the ground and simulating the ACK so we could get a continuous stream of packets, and using a separate "Error ACK" to resend only the garbled packets. It worked quite well in the simulations, achieving near hardwired transmittal rates. That was the closest I got to diving into the bowels of how things talk to each other.
Latest Article - A Concise Overview of Threads Learning to code with python is like learning to swim with those little arm floaties. It gives you undeserved confidence and will eventually drown you. - DangerBunny Artificial intelligence is the only remedy for natural stupidity. - CDP1802
Now add a torrent client on top of that, add meatballs, and serve.
I wanna be a eunuchs developer! Pass me a bread knife!
