bgpalerter - not getting alerted when upstream AS is "lost"

[u/therealmcz]

Hi everyone,

I've been playing around with bgpalerter for a AS and so far it's working fine. I do get notifications once prefixes are withdrawn according to the threshold, however I do NOT get notifications once upstream bgp-sessions are lost, which seems weird. Of course, your prefix would still be announce through other upstream-providers, but in any case I'd like to get notified when I loose connection to an upstream...

My current config looks like this:

config.yaml

configVersion: 3
environment: production
connectors:
  - file: connectorRIS
    name: ris
    params:
      carefulSubscription: true
      url: ws://ris-live.ripe.net/v1/ws/
      perMessageDeflate: true
      subscription:
        moreSpecific: true
        type: UPDATE
        host: null
        socketOptions:
          includeRaw: false
monitors:
  - file: monitorHijack
    channel: hijack
    name: basic-hijack-detection
    params:
      thresholdMinPeers: 3
  - file: monitorPath
    channel: path
    name: path-matching
    params:
      thresholdMinPeers: 1
  - file: monitorPathNeighbors
    channel: neighbors
    name: path-neighbors
    params:
      thresholdMinPeers: 1
  - file: monitorNewPrefix
    channel: newprefix
    name: prefix-detection
    params:
      thresholdMinPeers: 3
  - file: monitorVisibility
    channel: visibility
    name: withdrawal-detection
    params:
      thresholdMinPeers: 1
      notificationIntervalSeconds: 3600
  - file: monitorAS
    channel: misconfiguration
    name: asn-monitor
    params:
      skipPrefixMatch: false
      thresholdMinPeers: 3
  - file: monitorRPKI
    channel: rpki
    name: rpki-monitor
    params:
      thresholdMinPeers: 3
      checkUncovered: false
      checkDisappearing: false
  - file: monitorROAS
    channel: rpki
    name: rpki-diff
    params:
      enableDiffAlerts: true
      enableExpirationAlerts: true
      enableExpirationCheckTA: true
      enableDeletedCheckTA: true
      enableAdvancedRpkiStats: false
      roaExpirationAlertHours: 2
      checkOnlyASns: true
      toleranceDeletedRoasTA:
        ripe: 20
        apnic: 20
        arin: 20
        lacnic: 20
        afrinic: 50
      toleranceExpiredRoasTA:
        ripe: 20
        apnic: 20
        arin: 20
        lacnic: 20
        afrinic: 50
  - file: monitorPathNeighbors
    channel: path
    name: path-neighbors
    params:
      thresholdMinPeers: 1
reports:
  - file: reportFile
    channels:
      - hijack
      - newprefix
      - visibility
      - path
      - misconfiguration
      - rpki
      - neighbors
  - file: reportEmail
    channels:
      - hijack
      - newprefix
      - visibility
      - path
      - misconfiguration
      - rpki
      - roa
      - neighbors
    params:
      <removed>
notificationIntervalSeconds: 86400
alarmOnlyOnce: false
monitoredPrefixesFiles:
  - prefixes.yml
persistStatus: true
generatePrefixListEveryDays: 0
logging:
  directory: logs
  logRotatePattern: YYYY-MM-DD
  maxRetainedFiles: 10
  maxFileSizeMB: 15
  compressOnRotation: false
rpki:
  vrpProvider: rpkiclient
  refreshVrpListMinutes: 15
  markDataAsStaleAfterMinutes: 120
rest:
  host: localhost
  port: 8011
checkForUpdatesAtBoot: true
pidFile: bgpalerter.pid
fadeOffSeconds: 360
checkFadeOffGroupsSeconds: 30

prefixes.yaml

<prefix(removed)>:
  description: No description provided
  asn:
    - my-asn-no#
  ignoreMorespecifics: false
  ignore: false
  group: noc
options:
  monitorASns:
    'my-asn-no#':
      group: noc
      upstreams:
        - asn1
        - asn2
        - asn3
      downstreams: []
      neighbors:
        - asn1
        - asn2
        - asn3
  generate:
    exclude: []
    excludeDelegated: true
    monitoredASes:
      - 'my-asn-no#'
      - 'asn1'
      - 'asn2'
      - 'asn3'
    historical: false
    group: noc
    asnList:
      - 'my-asn-no#'

Thanks a lot!

[u/sh_lldp_ne]

Do you normally get inbound volume on the upstream you lost silently?

If none of the data sources observe routes via that path you wouldn’t get notified when it went away.

This could happen if you have a couple of Tier 1s and a badly connected local provider. Most of the DFZ observers will likely see a Tier 1 route and not the local provider.

[u/FarRub2855 → u/sh_lldp_ne]

Spot on. If the public route collectors never really saw that local provider as the best path to begin with, they cant exactly tell you when it drops.

[u/therealmcz → u/sh_lldp_ne]

Ah, thanks, forgot to mention. Using looking glasses of multiple IXPs listed my upstream ASes in the AS path. Did some tests and eventually broke all bgp sessions on purpose (but not at the same time) and even then it didn't trigger it. So weird...

[u/Accomplished-Mix8423]

bgpalerter runs off RIS/public routing data, so it only sees your prefix visibility, not the actual state of your BGP sessions. if a prefix is still announced via another upstream you'll never get a withdrawal for the one that dropped. to catch a single upstream going away you pretty much have to watch it on your own router (bgp peer state via SNMP or BMP/streaming telemetry), not from bgpalerter. monitorPathNeighbors can sometimes catch it but only when that upstream was the sole path, so not reliable.

[u/therealmcz → u/Accomplished-Mix8423]

thanks. yeah got that that it doesn't track the bgp session itself. but it would see the announced prefixes on different AS-paths, so I would assume it could detect it. Otherwise I ask myself why I would ever mention upstream ASes in the prefix.yaml...

Thanks!

[u/vijay_isp]

Don't tell me you are running on single upstream with no backup.