From time to time, you get an interesting support issue that requires more debugging and investigation than just checking the logs.
It all started with an escalation from a technical contact (not support) reaching out to the product team that a critical feature wasn't working for Canadian customers on our Progressive Web App (PWA). No reproduction steps, just the error:
Oops, there was an error. We're sorry, but we can't process your request right now. Please try again later.
The PWA is owned by our web team, but the component erroring out is now owned by our team. An IFrame wraps around our component.
Typically we have one rotating on-call developer that all support non-prod/prod issues funnel to each sprint. This was my sprint.
TLDR: Chrome SameSite update broke a critical feature. Took a year to find and fix. Should have set up components with the same domain bindings initially.
IIS Logs
What do we do when someone presents an error? Let's take a look at the logs for more information! As I start digging in and investigating, I usually note the details in notepad, the ticket item, and now Slack. I try to keep the notepad with scraps of information while the ticket item like a lab book. It turns out that the application logs are actually not consumed Splunk. The logs that are available after some chats are the IIS logs and the userId affected. The logs below are scrubbed:
2021-03-09 19:33:11 W3SVC2 MS10287 10.12.128.70 GET /errors/sorry.html - 443 - 99.230.49.18 HTTP/1.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/88.0.4324.190+Safari/537.36 https://somesite.com/?culture=en-CA&email=&yourMom=&hostUrl=https%3A%2F%2Fwtf.somesite.ca%2Fsecret-method%2Fadd%2Flegacy&isAddressTemporary=false&isSomeComponentTemporary=false&lastFour=&mode=Quantum&SomeComponentId=&SomeComponentIdHash=&postalCode=&returnUrl=https%3A%2F%2Fwtf.somesite.ca%2Fsecret-method&token=3d30fdb4-fe16-4b1e-8cc3-25419a2e28c6&userId=a96785cb-65a7-4439-a8ff-29357b3eaadc secureapp.somesite.com 200 0 0 5208 1274 0
2021-03-09 19:33:11 W3SVC2 MS10287 10.12.128.70 POST / culture=en-CA&email=&yourMom=&hostUrl=https%3A%2F%2Fwtf.somesite.ca%2Fsecret-method%2Fadd%2Flegacy&isAddressTemporary=false&isSomeComponentTemporary=false&lastFour=&mode=Quantum&SomeComponentId=&SomeComponentIdHash=&postalCode=&returnUrl=https%3A%2F%2Fwtf.somesite.ca%2Fsecret-method&token=3d30fdb4-fe16-4b1e-8cc3-25419a2e28c6&userId=a96785cb-65a7-4439-a8ff-29357b3eaadc 443 - 99.230.49.18 HTTP/1.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/88.0.4324.190+Safari/537.36 https://somesite.com/?culture=en-CA&email=&yourMom=&hostUrl=https%3A%2F%2Fwtf.somesite.ca%2Fsecret-method%2Fadd%2Flegacy&isAddressTemporary=false&isSomeComponentTemporary=false&lastFour=&mode=Quantum&SomeComponentId=&SomeComponentIdHash=&postalCode=&returnUrl=https%3A%2F%2Fwtf.somesite.ca%2Fsecret-method&token=3d30fdb4-fe16-4b1e-8cc3-25419a2e28c6&userId=a96785cb-65a7-4439-a8ff-29357b3eaadc secureapp.somesite.com 302 0 0 9505 5446 0
2021-03-09 19:32:29 W3SVC2 MS10287 10.12.128.70 GET /Main/GetValidationMessages url=%2F&culture=en-CA 443 - 99.230.49.18 HTTP/1.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/88.0.4324.190+Safari/537.36 https://somesite.com/?culture=en-CA&email=&yourMom=&hostUrl=https%3A%2F%2Fwtf.somesite.ca%2Fsecret-method%2Fadd%2Flegacy&isAddressTemporary=false&isSomeComponentTemporary=false&lastFour=&mode=Quantum&SomeComponentId=&SomeComponentIdHash=&postalCode=&returnUrl=https%3A%2F%2Fwtf.somesite.ca%2Fsecret-method&token=3d30fdb4-fe16-4b1e-8cc3-25419a2e28c6&userId=a96785cb-65a7-4439-a8ff-29357b3eaadc secureapp.somesite.com 200 0 0 175611 1169 15
2021-03-09 19:32:29 W3SVC2 MS10287 10.12.128.70 GET /static/css/sitemain.css - 443 - 99.230.49.18 HTTP/1.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/88.0.4324.190+Safari/537.36 https://somesite.com/?culture=en-CA&email=&yourMom=&hostUrl=https%3A%2F%2Fwtf.somesite.ca%2Fsecret-method%2Fadd%2Flegacy&isAddressTemporary=false&isSomeComponentTemporary=false&lastFour=&mode=Quantum&SomeComponentId=&SomeComponentIdHash=&postalCode=&returnUrl=https%3A%2F%2Fwtf.somesite.ca%2Fsecret-method&token=3d30fdb4-fe16-4b1e-8cc3-25419a2e28c6&userId=a96785cb-65a7-4439-a8ff-29357b3eaadc secureapp.somesite.com 200 0 0 9016 1085 0
2021-03-09 19:32:29 W3SVC2 MS10287 10.12.128.70 GET / culture=en-CA&email=&yourMom=&hostUrl=https%3A%2F%2Fwtf.somesite.ca%2Fsecret-method%2Fadd%2Flegacy&isAddressTemporary=false&isSomeComponentTemporary=false&lastFour=&mode=Quantum&SomeComponentId=&SomeComponentIdHash=&postalCode=&returnUrl=https%3A%2F%2Fwtf.somesite.ca%2Fsecret-method&token=3d30fdb4-fe16-4b1e-8cc3-25419a2e28c6&userId=a96785cb-65a7-4439-a8ff-29357b3eaadc 443 - 99.230.49.18 HTTP/1.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/88.0.4324.190+Safari/537.36 https://wtf.somesite.ca/ secureapp.somesite.com 200 0 0 8703 1204 31
The initial gets are loading of the form and the POST responded with a 302 (Found). The next entry is a is a GET to /error/sorry.html. So it looks like the POST redirected the IFrame to an error page. Not much to go on. Let's compare with the logs for a region that isn't broken like the US:
2021-03-09 22:30:08 W3SVC2 MS10290 10.12.128.69 GET /static/css/sitemain.css - 443 customer@gmail.com 72.214.228.333 HTTP/1.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/88.0.4324.190+Safari/537.36 https://secureapp.somesite.com/?culture=en-US&email=&firstSix=&hostUrl=https%3A%2F%2Fapp.somesite.com%2FsomeComponent-method%2Fadd%2Flegacy&isAddressTemporary=false&issomeComponentTemporary=false&lastFour=&mode=Account&someComponentId&someComponentIdHash&postalCode=&returnUrl=https%3A%2F%2Fapp.somesite.com%2FsomeComponent-method&token=&userId= secureapp.somesite.com 200 0 0 9016 3896 15
2021-03-09 22:30:08 W3SVC2 MS10290 10.12.128.69 POST / culture=en-US&email=&firstSix=&hostUrl=https%3A%2F%2Fwtf.somesite.com%2Fsecret-method%2Fadd%2Flegacy&isAddressTemporary=false&issomeComponentTemporary=false&lastFour=&mode=Quantum&SomeComponentId&SomeComponentIdHash&postalCode=&returnUrl=https%3A%2F%2Fwtf.somesite.com%2Fsecret-method&token=&userId= 443 customer@gmail.com 72.214.228.333 HTTP/1.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/88.0.4324.190+Safari/537.36https://secureapp.somesite.com/?culture=en-US&email=&firstSix=&hostUrl=https%3A%2F%2Fwtf.somesite.com%2Fsecret-method%2Fadd%2Flegacy&isAddressTemporary=false&issomeComponentTemporary=false&lastFour=&mode=Quantum&SomeComponentId&SomeComponentIdHash&postalCode=&returnUrl=https%3A%2F%2Fwtf.somesite.com%2Fsecret-method&token=&userId= secureapp.somesite.com 200 0 0 5987 7839 1733
...
Interesting to note that the en-CA has more information right in the query parameters. Also logging of email address shouldn't occur. I suspect it could just be an IIS field to remove. Also interesting to note that the en-CA referes to wtf.somesite.com instead of wtf.somesite.ca. A cross-domain call in an IFrame...
Application Logs
An SRE was able to get the application logs as a file off the box and it goes back 9 months. Looking at the most recent logs:
3/9/2021 12:26:32 AM General Information [] - FormConfigServiceFacade: Error Field FirstName type is Uknown
3/9/2021 12:28:26 AM General Information [] - FormConfigServiceFacade: Error System.Runtime.Serialization.SerializationException: Unable to deserialize XML body with root name 'OperationResultOfPostalCodeLookupDataqutz94KU' and root namespace 'http://somesite.com/contentservices/common' (for operation 'PostalCodeLookup' and contract ('IFormConfigService', 'http://tempuri.org/')) using DataContractSerializer. Ensure that the type corresponding to the XML is added to the known types collection of the service.
Server stack trace:
at System.ServiceModel.Dispatcher.SingleBodyParameterMessageFormatter.ReadObject(Message message)
at System.ServiceModel.Dispatcher.SingleBodyParameterDataContractMessageFormatter.ReadObject(Message message)
at System.ServiceModel.Dispatcher.DemultiplexingClientMessageFormatter.DeserializeReply(Message message, Object[] parameters)
at System.ServiceModel.Dispatcher.ProxyOperationRuntime.AfterReply(ProxyRpc& rpc)
at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at XYZ.WCF.Contract.IFormConfigService.PostalCodeLookup(String postalcode, String country)
at XYZ.Web.UI.Payment.FormConfig.FormConfigServiceFacade.GetPostalCodeLookupData(UserAddressModel address) in d:\B\54\29733\Sources\Web.UI\FormConfig\FormConfigServiceFacade.cs:line 110
3/9/2021 2:26:49 AM General Information [] - FormConfigServiceFacade: Error Field FirstName type is Uknown
3/9/2021 4:28:21 AM General Information [] - FormConfigServiceFacade: Error Field FirstName type is Uknown
3/9/2021 6:28:59 AM General Information [] - FormConfigServiceFacade: Error Field FirstName type is Uknown
3/9/2021 8:40:35 AM General Information [] - FormConfigServiceFacade: Error Field FirstName type is Uknown
3/9/2021 10:47:24 AM General Information [] - FormConfigServiceFacade: Error Field FirstName type is Uknown
3/9/2021 11:10:24 AM General Information [] - FormConfigServiceFacade: Error Field FirstName type is Uknown
3/9/2021 1:10:29 PM General Information [] - FormConfigServiceFacade: Error Field FirstName type is Uknown
3/9/2021 3:11:28 PM General Information [] - FormConfigServiceFacade: Error Field FirstName type is Uknown
3/9/2021 3:51:04 PM General Information [] - FormConfigServiceFacade: Error System.Runtime.Serialization.SerializationException: Unable to deserialize XML body with root name 'OperationResultOfPostalCodeLookupDataqutz94KU' and root namespace 'http://somesite.com/contentservices/common' (for operation 'PostalCodeLookup' and contract ('IFormConfigService', 'http://tempuri.org/')) using DataContractSerializer. Ensure that the type corresponding to the XML is added to the known types collection of the service.
Hm, possible failure when looking up postal code? What is up with this OperationResultOfPostalCodeLookupDataqutz94KU. Nice typos. Some of these can be reduced with a Spell Checker plugin .
Source Code
While our team owns this codebase, this was only something we've recently taken on and nobody is familiar with this code on our current team. The original developers have moved to other teams or left the company.
The first challenge was tracking down the source code. I vaguely remember it being moved to Github Enterprise and did a search there for the FormConfigServiceFacade.cs. This did bring up a repository that contained the source.
I also got in contact with some of the original devs and it turns out that the latest up to date source was kept in Team Foundation Server (TFS) which has been deprecated internally. I was surprised that it is still accessible.
Tracing the error back to the actual client calling the PostalCodeLookup from the FormConfigServiceFacade.cs, I can see the URI that is used to make this lookup. I also see references to fallback URIs for the same functionality, but there isn't actually fallback logic to use those.
Third Party API
I start digging around the postal code lookup code and the API. Using the hard coded credentials in the code, I was able to perform GETs via the browser with valid responses given the postal/zip codes. I started digging further into the site and noticed several things:
- The login page isn't using TLS/SSL.
- Robots.txt is accessible and I can see where denies are registered.
- The search box may be vulnerable to SQL injection.
- The login page may be vulnerable to SQL injection.
Hitting the demo API link provided:
{"status":{"message":"the hourly limit of X credits for demo has been exceeded. Please use an application specific account. Do not use the demo account for your application.","value":19}}
I was able to actually get the payload I expect by changing the query parameter in URI from demo to something easily guessable and boom, a successful response with the expected details.
This API doesn't provide any sensitive information. It has many features for address verification and such. However, looking at the About page, one can see all the other sites that rely on this API. I would consider this a weak point and considered at the time of creation Google APIs may not have been available or maybe the company didn't want to shell out any associated costs.
I had suspected that perhaps something in the response payload from the site had changed or that a parameter was possibly too long, but after some testing with a Canadian postal code that yielded a short city name, I ruled this out.
The deserialization issue was coming from a WCF service that was calling this API. The application logs may be leading me astray.
Back to the Drawing Board
Going back to the IIS logs, I decided to see if I can just hit the component directly instead of through the PWA/IFrame. I was able able to do so and was able to successfully conduct the action that was yielding the error. I didn't get a confirmation, just an never-ending progress bar. I was only able to know it succeeded by looking at the IIS Logs and the logging out and back into the site.
Good new, it likely isn't an issue with the component code.
Web Team
I had already been in contact with the web devs and working with the on-call developer. I was investigating the component while he was investigating any issues on the PWA side. After stating that I could conduct a successful transaction through the component directly, he dove deeper into his domain. It turns out that the functionality isn't broken if the PWA is rendered through Firefox. The functionality is broken is for Chrome, Safari, Edge, and possibly others.
Interesting...
Looking at the IIS Logs to see how long this has been happening and filter for en-CA and Chrome browsers:
The the errors only go back to mid February 2020.
Let's see what happened around this time with Chrome...
Google: February 2020 Chrome: blog.chromium.org/2020/02/samesite-cookie-c..
If it was a change to how Chrome and other browsers handle SameSite cookies, then that explains the lack of errors prior to February (build 80) and the upwards trend of errors until April where we see a decline. Turns out that there was a rollback on the same site cookies changes and then a re-release of the changes in May (build 84). After July we see that the error trend stays stable and still occurs with some spikes, but not to the extent to the Feb-June timeframe. Possibly due to users giving up on that functionality or using Firefox.
Reproduction Steps
We have some clues that this could be, but let's find out exactly which cookie is causing the issue. I've had some previous experience with he workflow that was mentioned, but still no solid reproduction steps. Took and educated guess while Chrome and Firefox developer tools were open (F12 - Alternatively BurpSuite and Fiddler). Below are the request headers for the POST:
Chrome:
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Cache-Control: max-age=0
Connection: keep-alive
Content-Length: 3728
Content-Type: application/x-www-form-urlencoded
Cookie: notice_gdpr_prefs=0,1,2:; notice_preferences=2:; cmapi_gtm_bl=; cmapi_cookie_privacy=permit 1,2,3; lt-anonymous-id="0.ad67ddb11781ebb0660"; lt-session-data={"id":"0.cdad9a41781ebb0661","lastUpdatedDate":"2021-03-11T00:41:16Z"}; lt-pageview-id="0.155538f51781ebb0666"; notice_behavior=implied,us
Host: secureapp.somesite
Origin: https://secureapp.somesite
Referer: https://secureapp.somesite/?culture=en-CA&email=&firstSix=&hostUrl=https%3A%2F%2Fwtf.somesite.ca%2FsomeComponent-method%2Fadd%2Flegacy&isAddressTemporary=false&issomeComponentTemporary=false&lastFour=&mode=Account&someComponentId=&someComponentIdHash=&postalCode=&returnUrl=https%3A%2F%2Fwtf.somesite.ca%2FsomeComponent-method&token=3d30fdb4-fe16-4b1e-8cc3-25419a2e28c6&userId=a96785cb-65a7-4439-a8ff-29357b3eaadc
Sec-Fetch-Dest: iframe
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: same-origin
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.190 Safari/537.36
Firefox:
Host:secureapp.somesite
User-Agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:86.0) Gecko/20100101 Firefox/86.0
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Language:en-US,en;q=0.5
Accept-Encoding:gzip, deflate, br
Content-Type:application/x-www-form-urlencoded
Content-Length:4190
Origin:https://secureapp.somesite
Connection:keep-alive
Referer:https://secureapp.somesite/?culture=en-CA&email=&firstSix=&hostUrl=https%3A%2F%2Fwtf.somesite.ca%2FsomeComponent-method%2Fadd%2Flegacy&isAddressTemporary=false&issomeComponentTemporary=false&lastFour=&mode=Account&someComponentId=&someComponentIdHash=&postalCode=&returnUrl=https%3A%2F%2Fwtf.somesite.ca%2FsomeComponent-method&token=26605DBEDBEDB386472E425AB40EA5810AAC8E19&userId=C3D88340-A729-4D1C-A608-A7F22B2902F3
Cookie:optimizelyEndUserId=oeu1615528825254r0.31460353898326143; _gcl_au=1.1.121772586.1615528826; _ga=GA1.2.468030874.1615528826; _gid=GA1.2.725382376.1615528826; notice_preferences=2:; notice_gdpr_prefs=0,1,2:; cmapi_cookie_privacy=permit 1,2,3; tiWQK2tY=AIqYBSV4AQAA-iA-wQoBKIjczLH0y4IKJY5gq2cBn0mulySp7BplA-GK6q1W|1|1|287eb27e8a6ec6a76366c64616402e402ee7d54e; cmapi_gtm_bl=; TS01d169da=014914009638ae182acf562351ea9bf471b6a671e8a3e749ed5714b621d056b5b57188da04afb5ea94799b94a1b34b8a969241cdde349ef541f690a119365294900561428301ae850fe316acf4a8cc300fef93ca3d7049b7ff36cc8929a79f1f1e92468ee4f4689863024003b4177c1be1048edbbf3991e08c8661693b76fb5de1da4006f9817fd51bc00edeb5b27b9e9c4205c276; notice_behavior=implied,us; __RequestVerificationToken=X-adM5XewDo2fKijYgU6Gs1fopkSNq-nYTLfMkDPzOzjeI9NLBozS1P_lJMslPosyuO-w6fkTIYtXhZIQjT5jV7fkC1VyvnXYSzoZNCFAHA1; TS013475f5=013fe7004e1abe3e52b351e99be950a0760a4ad7110be6c71691388f5c44a935a31c31180297e38468159cd45b7a890763300c44ee25b0cd8ace852e82146dcc19efd582a6; _uetsid=3f385b8082f811ebbf59d3bdfe7be3eb; _uetvid=3f38b89082f811eb9a7c7724c536f5a0
Upgrade-Insecure-Requests:1
Looks like the we're missing the RequestVerificationToken. We can see in Chrome that the cookie gets blocked with the following message:
This cookie didn't specify a "SameSite" attribute when it was stored and was defaulted to "SameSite=Lax", and was blocked because the request was made from a different site and was not initialized by a top-level navigation. The cookie had to have been set with "SameSite=None" to enable cross-site usage.
This token is set from the initial load of the form in the IFrame. The RequestVerificationToken is used to prevent Cross-Site Request Forgery (CSRF). This token in the cookie ensures that the form submission (POST) in the request matches with the token in the cookie/header.
After using EditThisCookie (also can replay using Burp Suite/Fiddler after editing the request) and specifying the domain=somesite.ca, I was able to complete the transaction through Chrome!
Next Steps
Now we know the root cause we can decide on how to fix this.
1) We could make a code change in the component by specifying that the RequestVerificationToken with the either the domain or SameSite=None.
2) We could add bindings to secureapp.somesite.ca so that we have parity with the en-US site that seems to inherently trust the domain of somesite.com.
Option two is the right option since we shouldn't be having these cross-domain calls. It turns out there were additional workarounds to get this to work originally that should be removed.
Either way we should also be prepared to make code changes and validate in a test environment.
Obstacles
As I mentioned previously this source code is in TFS and the build agents and controllers have been disabled. Looks like getting it to build is not an easy task. Long term efforts would be to migrate this to the new source control and create a new definition in a new build pipeline.
After some effort (adding the .ca domain to the web.config) we were able to deploy this to a test environment and once customErrors was disabled:
<configuration>
<system.web>
<customErrors defaultRedirect="YourErrorPage.aspx"
mode="off">
<error statusCode="500"
redirect="InternalErrorPage.aspx"/>
</customErrors>
</system.web>
</configuration>
We can see:
500 HttpAntiForgery The required anti-forgery cookie __RequestVerificationToken" is not present.
Nice parity with production and we can test both code fixes and infrastructure changes.
Once the fix was validated in the test environment we were able to move it to production and validate with the help of our Web Software, Network, System, and Site Reliability engineers.
Summary
Functionality was broken due to Chrome 80 and 84 builds. Been about a year before we had this reported and fixed. There are still are still numerous of cookies blocked due to SameSite not being specified. I'm surprised that we have no other issues, but maybe they just aren't reported. Possibly analytics/metrics have been impacted.
After release of all infrastructure/server and additional code changes we can see the 302 redirects trend down. All additional tech debt items (migration of the source code for the component with CI/CD pipeline) tracked, but not yet reviewed/prioritized. Awesome cross-team collaboration to get this out.
I remember hearing possible issues with this Chrome update a year ago via the SANs StormCast and Security Now Podcast . I never expected this to hit me since the front-end isn't within my domain.
On top of this issue we had a network change that impacted other production services that kept me from a good night's rest.
When it rains...
Key Takeaways
Tech Debt needs to be paid down. Somethings slip due to team changes and others due to business demands. Not taking care of it is like a gamble that company takes. Will it cost more money/customer impact to do it now or later? Coding is more like tending a garden than building bridge. In our case we need to migrate the source code over for the component, working build, and deployment pipeline.
Have automated smoke tests. The en-US site had all automation pass. There was no automation for en-CA.
Have proper monitoring and alerts setup. Have an easy to glance at dashboard based off IIS and application log details. Have alerts set with the appropriate threshold triggers.
Information Exposure: Don't log more information than you need in logs. Layered logging approach (RegEx filtering, Destructurama Attributed, wrapper around log frameworks. Tune IIS field logging.
Log enough information to debug. Easier said than done sometimes.
Be careful of your third-party APIs. Same can be said for third-party libraries.
Pay attention to error trends.
Attempt reproduction of front-end issues with different browsers.
Web debugging uses the same tools as Web CTFs). Getting experience with CTFs is helpful for developers to see security risks and write more robust code. Besides just reading/watching OWASP Top 10 articles/videos, your knowledge and is solidified by using the same tools that hackers. Be a better defender by being a attacker/hacker.
Don't hard code credentials/configuration.
Sleep is very very important.