Had some errors reported from an on-premises service attempting to publish events to an Azure Eventhub in our test environments:
System.UnauthorizedAccessException: Ip has been prevented to connect to the endpoint.
For more information see:
Virtual Network service endpoints:
Event Hubs: https://go.microsoft.com/fwlink/?linkid=2044192
Service Bus: https://go.microsoft.com/fwlink/?linkid=2044235
IP Filters:
Event Hubs: https://go.microsoft.com/fwlink/?linkid=2044428
Service Bus: https://go.microsoft.com/fwlink/?linkid=2044183
TrackingId:3445ee04-4a45-42bb-ad5b-821033b57e58_G7, SystemTracker:AmqpGatewayProvider, Timestamp:2021-05-03T22:55:50
We had recently re-enabled the firewall for this Eventhub. Looks like it was disabled by someone previously and our service had whitelisted IP addresses in the firewall configuration.
Diagnostic logging can be enabled and the the following Kusto query will show all denied connections for the last 7 days:
AzureDiagnostics
| where TimeGenerated > ago(7d)
| where ResourceProvider =="MICROSOFT.EVENTHUB"
| where Category == "EventHubVNetConnectionEvent"
| where action_s == "Deny Connection"
If you have multiple resources (KeyVaults, EventHubs, etc) in different environments, you can narrow it down with
| where Resource == 'your_specific_resource'
The client IP address (IPAddress) will be one of the returned fields. The client IP denied by the firewall had the same first three octets as some of the IPs already whitelisted:
Example:
78.88.134.*
I suspect that the external IPs are rotating within some allotted range. To confirm, we can check the server that is having the connection issue. Sadly, it's always a hassle to use external tools or the browser since the servers are locked down to some extent (they should be).
The external IP address can be check by the following hitting an external site/API:
curl ifconfig.io
If on Windows Servers and curl isn't available:
((Invoke-WebRequest -uri ifconfig.io).Content -Match $regex)|Out-Null; $Matches[0]
Alternatively, the ParsedHtml property could be used similar to the DOM in Javascript to find what you're looking for.
The server IP Address does match and we're granting all IP ranges from 1-254 via CIDR notation for now.
78.88.134.1/24
This leaves that range open, but better than not having the firewall enabled...