Calculate the average ignoring a specific filter

Hello, I'm having trouble calculating the total average of payment days. The situation is as following:

I have 4 tables that are connected to my calculation:

1. 'Invoice History': stores all the invoices with an InvoiceNr, debtorNr and an Invoice_Date
2. 'Payment Behaviour': stores an InvoiceNr and the PaymentDays (= a number that indicates te days between the invoice_date and the payment date)
3. 'Debtors': Stores the information of all the debtors
4. 'Dates': A table with all dates which are relevante to my report.

I have already made a measure that calculates the average payment days of all the individual debtors:

Average PaymentDays = 
    CALCULATE (
        AVERAGE ( 'Payment Behaviour'[PaymentDays] );
        FILTER (
            'Invoice History';
            RELATED ( 'Payment Behaviour'[InvoiceNr] ) = 'Invoice History'[InvoiceNr]
        )
    )

I used a filter because not all the invoices in the 'invoice history' table are paid yet. So I looked for my calculation only on the paid invoices. Now I want to make a measure that calculates the average payment days of all the debtors combined ignoring any debtor that I've selected in a slicer. I tried a lot of different things already but I can't figure out what I'm doing wrong. Every time I make a measure it gives back the average payment days of alle the debtors ignoring even the months that are selected. So I get back the same average for every period. I just want to ignore the selected debtors. Not the selected dates. 

This is the last thing I tried:

Average PaymentDays Overall = 
    CALCULATE (
        AVERAGE ( 'Payment Behaviour'[PaymentDays] );
        FILTER (
            ALLEXCEPT('Invoice History'; Dates[date]);
            RELATED ( 'Payment Behaviour'[InvoiceNr] ) = 'Invoice History'[InvoiceNr]
        )
    )

Thanks in advance!