Find a solution for automating a manual task for searching a TFS change-set dot-net technology
-
Hi,
Got a scenario: we have ASMX web services and we auto-test the service with the help of Soap UI Tool (by SmartBear) every night after launching a new build which also happens every night after merging all the latest changes in the TFS source codes. In the process of test step’s execution, some test step gets failed due to mismatch in test parameter values; this mismatch might be due to change in the values in the source code and check-in by the developer. Some test step gets failed due to null reference and some due to mismatch in testing value stored in sqlserver database. When we try to find the reason of failed test step, it’s a time consuming task every day to find the change-set which caused the failure. Therefore, we want to automate the process to get the change-set for the failed test steps. Here, looking for your expert suggestion for developing a solution for the same.
Please let me know in case you need further clarification.
Thank youThank you JP
-
Hi,
Got a scenario: we have ASMX web services and we auto-test the service with the help of Soap UI Tool (by SmartBear) every night after launching a new build which also happens every night after merging all the latest changes in the TFS source codes. In the process of test step’s execution, some test step gets failed due to mismatch in test parameter values; this mismatch might be due to change in the values in the source code and check-in by the developer. Some test step gets failed due to null reference and some due to mismatch in testing value stored in sqlserver database. When we try to find the reason of failed test step, it’s a time consuming task every day to find the change-set which caused the failure. Therefore, we want to automate the process to get the change-set for the failed test steps. Here, looking for your expert suggestion for developing a solution for the same.
Please let me know in case you need further clarification.
Thank youThank you JP
A time consuming process? What, you mean debugging the code? Yeah, you're not going to automate that process.
A guide to posting questions on CodeProject
Click this: Asking questions is a skill. Seriously, do it.
Dave Kreskowiak -
Hi,
Got a scenario: we have ASMX web services and we auto-test the service with the help of Soap UI Tool (by SmartBear) every night after launching a new build which also happens every night after merging all the latest changes in the TFS source codes. In the process of test step’s execution, some test step gets failed due to mismatch in test parameter values; this mismatch might be due to change in the values in the source code and check-in by the developer. Some test step gets failed due to null reference and some due to mismatch in testing value stored in sqlserver database. When we try to find the reason of failed test step, it’s a time consuming task every day to find the change-set which caused the failure. Therefore, we want to automate the process to get the change-set for the failed test steps. Here, looking for your expert suggestion for developing a solution for the same.
Please let me know in case you need further clarification.
Thank youThank you JP
Member 1263842 wrote:
find the change-set which caused the failure.
Since you check in code every day and run this every night then the change-set that caused the error is one of the ones from that day. I guess I don't quite follow what you are asking.
There are two kinds of people in the world: those who can extrapolate from incomplete data. There are only 10 types of people in the world, those who understand binary and those who don't.
-
Hi,
Got a scenario: we have ASMX web services and we auto-test the service with the help of Soap UI Tool (by SmartBear) every night after launching a new build which also happens every night after merging all the latest changes in the TFS source codes. In the process of test step’s execution, some test step gets failed due to mismatch in test parameter values; this mismatch might be due to change in the values in the source code and check-in by the developer. Some test step gets failed due to null reference and some due to mismatch in testing value stored in sqlserver database. When we try to find the reason of failed test step, it’s a time consuming task every day to find the change-set which caused the failure. Therefore, we want to automate the process to get the change-set for the failed test steps. Here, looking for your expert suggestion for developing a solution for the same.
Please let me know in case you need further clarification.
Thank youThank you JP
If I were in your team, I would suggest that you would be better off with a comprehensive CI process with decent unit tests and pre-flight check ins to ensure that you don't regularly commit broken code.It's better to prevent problems than look to apportion blame.
This space for rent
-
Hi,
Got a scenario: we have ASMX web services and we auto-test the service with the help of Soap UI Tool (by SmartBear) every night after launching a new build which also happens every night after merging all the latest changes in the TFS source codes. In the process of test step’s execution, some test step gets failed due to mismatch in test parameter values; this mismatch might be due to change in the values in the source code and check-in by the developer. Some test step gets failed due to null reference and some due to mismatch in testing value stored in sqlserver database. When we try to find the reason of failed test step, it’s a time consuming task every day to find the change-set which caused the failure. Therefore, we want to automate the process to get the change-set for the failed test steps. Here, looking for your expert suggestion for developing a solution for the same.
Please let me know in case you need further clarification.
Thank youThank you JP
It seems to me that you could tackle this in one of two ways: (1) Finer granularity on your build process. Somebody else already suggested a full-blown CI solution, but you could (I suppose) build say 4 times per day, which would dice-up the problem and presumably help; (2) Semi-automate the debug process by correlating changed modules with locations where errors are arising. Option 1 requires that you can get those merges done more often and turn the build handle. Option 2 strikes me as a good thing to do anyway, but whether it adds much value will obviously depend on how "orthogonal" the changes are that people are making to the code: if many are making changes in the same few files then it's not really going to work.
-
A time consuming process? What, you mean debugging the code? Yeah, you're not going to automate that process.
A guide to posting questions on CodeProject
Click this: Asking questions is a skill. Seriously, do it.
Dave KreskowiakHi Dave, Soap UI tool generates log file after each execution. To know the reason for the failure, we go through log file. Log file does not have a proper stack trace with filename, methodname, line number etc for the failed test rather the generic reason for the failure of the test step. So, we have to debug the application for the failed method and this process takes a good amount of time to debug and find the bug. Hope it clarifies the question. Thanks
-
If I were in your team, I would suggest that you would be better off with a comprehensive CI process with decent unit tests and pre-flight check ins to ensure that you don't regularly commit broken code.It's better to prevent problems than look to apportion blame.
This space for rent
-
It seems to me that you could tackle this in one of two ways: (1) Finer granularity on your build process. Somebody else already suggested a full-blown CI solution, but you could (I suppose) build say 4 times per day, which would dice-up the problem and presumably help; (2) Semi-automate the debug process by correlating changed modules with locations where errors are arising. Option 1 requires that you can get those merges done more often and turn the build handle. Option 2 strikes me as a good thing to do anyway, but whether it adds much value will obviously depend on how "orthogonal" the changes are that people are making to the code: if many are making changes in the same few files then it's not really going to work.
-
You're welcome.
This space for rent
-
Hi Dave, Soap UI tool generates log file after each execution. To know the reason for the failure, we go through log file. Log file does not have a proper stack trace with filename, methodname, line number etc for the failed test rather the generic reason for the failure of the test step. So, we have to debug the application for the failed method and this process takes a good amount of time to debug and find the bug. Hope it clarifies the question. Thanks
In this case, I agree with Pete. You're not going to be able to write code to automatically find which check in caused the problem. You could even have multiple problems. You have two possible paths here. The first is to debug the code and do what you've been doing. This is, obviously, time consuming and reactionary. The other is to implement a check in process where the code is tested before you allow it to be checked in. It sounds like you've got people "breaking the build" on just about every check in. That's not tolerable and must be prevented. Yeah, it sounds like it's more time to do this. It is but instead of spending that time on finding and fixing the bug you're spending it on preventing the bugs and improving the quality of the code.
A guide to posting questions on CodeProject
Click this: Asking questions is a skill. Seriously, do it.
Dave Kreskowiak -
Thank you, Nick for sharing your thought. Which part of build process, I should focus? Yes, there is possibility of multiple check-ins on a file.
By "finer granularity" I meant only that you could build more frequently. For example if currently you build at (say) midnight each day after everybody has finished work, then if Module "X" was changed 3 times and now fails to compile you've got to figure out which change caused it to fail. If, however, you'd done several builds during the day (e.g. every 2 hours) then you have more chance of discovering a broken build for which only one checkin has been done. Taking this to an extreme you end up with CI = Continuous Integration in which typically there'll be a gate on the checkin process with some mandatory regression tests needing to be passed before checkin can be done. There are lots of ways to implement this in practice: you could allow checkin but advance a stable label only as each checkin is built and validated. In the ideal world CI should ensure that your code always builds. However this is a big topic which is why I suggested only a small change to your existing process rather than a fundamental change of strategy. Do you know Jenkins? If not, it's worth reading about, including [Jenkins CI](Continuous integration with Jenkins - Tutorial).
-
In this case, I agree with Pete. You're not going to be able to write code to automatically find which check in caused the problem. You could even have multiple problems. You have two possible paths here. The first is to debug the code and do what you've been doing. This is, obviously, time consuming and reactionary. The other is to implement a check in process where the code is tested before you allow it to be checked in. It sounds like you've got people "breaking the build" on just about every check in. That's not tolerable and must be prevented. Yeah, it sounds like it's more time to do this. It is but instead of spending that time on finding and fixing the bug you're spending it on preventing the bugs and improving the quality of the code.
A guide to posting questions on CodeProject
Click this: Asking questions is a skill. Seriously, do it.
Dave Kreskowiak -
By "finer granularity" I meant only that you could build more frequently. For example if currently you build at (say) midnight each day after everybody has finished work, then if Module "X" was changed 3 times and now fails to compile you've got to figure out which change caused it to fail. If, however, you'd done several builds during the day (e.g. every 2 hours) then you have more chance of discovering a broken build for which only one checkin has been done. Taking this to an extreme you end up with CI = Continuous Integration in which typically there'll be a gate on the checkin process with some mandatory regression tests needing to be passed before checkin can be done. There are lots of ways to implement this in practice: you could allow checkin but advance a stable label only as each checkin is built and validated. In the ideal world CI should ensure that your code always builds. However this is a big topic which is why I suggested only a small change to your existing process rather than a fundamental change of strategy. Do you know Jenkins? If not, it's worth reading about, including [Jenkins CI](Continuous integration with Jenkins - Tutorial).