AFM assessment of order execution quality on PFOF trading venues

Assessing the quality of executions on trading venues The “Comparative Pricing Model” March 2022 (version 2) AFM

2 Assessing the quality of executions on trading venues Contents Summary 3 1 Introduction 4 2 Methodology 5 3 Research outcomes 7
Annex I Specifications of the Comparative Pricing Model 17
Annex II Questions and answers 20
Annex III The Comparative Pricing Model and order-data
analysis 23

3 Assessing the quality of executions on trading venues

Summary
Following the emerging worldwide public debate on the risks and presumed benefits of the practice of payment for order flow (PFOF) 1 , the European Securities and Markets Authority (ESMA) published a warning on the risks arising from PFOF. PFOF causes a conflict of interests between a firm and its clients as it incentivizes the firm to choose the third party offering the highest payment, rather than the best possible outcome for its clients when executing or routing their orders for execution. Therefore, ESMA called on National Competent Authorities (NCAs) to dive deeper into the issue. Subsequently, the Dutch Authority for the Financial Markets (AFM) looked into the execution quality of two PFOF trading venues and one non-PFOF trading venue (all three used by pan-European operating low-cost neo￾brokers) as well as one low-cost investment firm. The AFM selected these trading venues and low-cost investment firm based on criteria such as data availability and a substantial presence of activities in multiple European countries. The analyzed trading venues are used by neo-brokers with comparably low commissions. The results of our analysis show that the PFOF trading venues structurally offered worse execution prices compared to real transactions happening at a similar point in time on multiple other trading venues. The CNMV applied the same methodology to a different dataset and obtained similar results.2 In order to assess execution quality, the AFM developed a methodology which provides a robust indicator of a trading venue’s execution quality based on post-trade data: the Comparative Pricing Model. The methodology is easy to replicate by other NCAs using their own available datasets. Replication was a key concern for us, and – since not all NCAs have order-data readily available – one of the primary reasons why we chose a post-trade data-based methodology. The AFM applies the Comparative Pricing Model to analyze how execution prices on one trading venue compares to execution prices on other trading venues. In its analysis, the AFM considers the price of a transaction to be better when the client is selling at a higher price (or buying at a lower price) than the price of any transaction on any reference trading venue in the same instrument in the same second. Similarly, the AFM considers a trade to be worse priced when the client is selling at a price lower (or buying at a price higher) than the price of any transaction on any of the reference trading venues in the same instrument in the same second. If neither is the case, the execution price is considered of similar quality. The results show that for the two PFOF trading venues, most retail client transactions are executed at a worse price compared to the most liquid reference markets. For most of the transactions (68-72% for PFOF trading venue X and 81-83% for PFOF trading venue Y) the execution price was worse. On PFOF trading venue X the average price deterioration for a transaction of € 3,000
is € 1.44, and € 3.46 for PFOF trading venue Y. For the third trading venue (Z), a non-PFOF trading venue, most of the retail client transactions are executed at a similar price (74-77%) compared to the reference markets, with the average price deterioration for a trade of € 3,000 being € 0.24. For the investment firm we examined, the percentage worse, better or similar executions are almost evenly divided, with the average price deterioration for a transaction of € 3,000 being € 0.42. In response to version 1 of this paper, the AFM noticed that some parties would have liked us to compare transaction-data with order-data (instead of with transaction-data), some of these respondents claimed such a 1 Payment for order flow is the practice of a third party such as a regulated market, market maker or liquidity provider paying any monetary or non-monetary benefits to an investment firm for routing their clients’ orders to that third party for execution. 2 CNMV, ‘Payment for Order Flow: an analysis of the quality of execution of a zero-commission broker on Spanish stocks’, 15 February
2022. Link: https://www.cnmv.es/DocPortal/Publicaciones/OTROS/Analisis_PFOF.pdf.

4 Assessing the quality of executions on trading venues method might provide different results or a possibly better method to assess execution quality. Therefore, in the March 2022 update of this paper, the AFM added an analysis where we use pre-trade (or “order”) data to compare quality of execution. The results are very similar to the results obtained via the Comparative Pricing Model which uses post-trade data. That is to say: the PFOF venues structurally underperform compared to quoted3 prices as well as actual execution prices. This confirms our belief that the Comparative Pricing Model by itself functions as a robust indicator of a trading venue’s or investment firm’s execution quality. Additional analyses and refinements of the methodology would provide broader insights into order execution quality within the EU. 1 Introduction After the GameStop debacle, a public debate emerged on the practice of PFOF, which causes a conflict of interest between a firm and its clients as it incentivizes the firm to choose a third party offering the highest payment, rather than the best possible outcome for its clients, when executing or routing clients’ orders. In July 2021, ESMA warned investors for the risks arising from PFOF and called on NCAs to dive deeper into the issue. 4 The AFM has analyzed the execution quality on two PFOF trading venues, one non-PFOF trading venue and one low-cost investment firm. The trading venues and investment firms were chosen based on criteria such as data availability and a substantial presence of activities in multiple European countries. In the March 2022 update of this paper the AFM added annex III:

• An analysis where we compare actual execution prices on the three trading venues with the relevant
  pre-trade data (best bid and best offer at the exact millisecond) on a liquid reference market. The public attention on PFOF also put a spotlight on the difficulties in establishing a proper reference price, which is critical to the assessment of the quality of order execution. A reference price is not the only component that is required for the assessment of execution quality for retail orders, as fees for clearing and settlement, connectivity and financial infrastructure should be included in such assessments as well. Most retail investors are laypersons when it comes to order execution, let alone capable to influence the execution quality of their orders. In the current MiFID II best execution framework, the responsibility to substantiate the decision for (a) particular execution venue(s) lies primarily with the investment firm.5 The approach to assess and achieve best execution varies per type of financial instrument. 6 In this paper we focus on a method to create a reliable reference price to allow for an accurate assessment of the price of execution, a key component in the assessment of any broker’s compliance with best execution requirements for retail client orders. To assess the price of execution of shares, the AFM developed a method that:
• makes use of data that is readily available for NCAs
• can be applied to different kinds of trading venues (for instance single market maker vs. multiple
  market maker venues)
  3 The AFM used the best bid and best offer at the millisecond of the transaction on Euronext Amsterdam for this analysis. 4 https://www.esma.europa.eu/press-news/esma-news/esma-warns-firms-and-investors-about-risks-arising-payment-order-flow 5 Article 66 (2) Delegated Regulation (EU) 2017/565. 6 Recital 104 Delegated Regulation (EU) 2017/565: “(…) given the differences in market structures or the structure of financial instruments, it may be difficult to identify and apply a uniform standard of and procedure for best execution that would be valid and effective for all classes of instrument. Best execution obligations should therefore be applied in a manner that takes into account the different circumstances associated with the execution of orders related to particular types of financial instruments.”

5 Assessing the quality of executions on trading venues

• is easy to understand and
• is easy to replicate by other NCAs.
  This resulted in a method that uses post-trade data to compare the execution prices of shares on one trading venue to the prices of execution on multiple other trading venues. In particular, the method shows whether a client’s order would have been better off when executed on another European trading venue, assuming these prices would have been available as well to the broker’s clients at the relevant timestamps. We elaborate on the methodology in chapter 2. In chapter 3 we show the results of applying the Comparative Pricing Model to three trading venues and one investment firm. In Annex I the methodology is explained step by step. Annex II provides Q&As about the background, limitations and opportunities of the Comparative Pricing Model. Annex III7 shows how the Comparative Pricing Model compares to an analysis based on pre￾trade (“order”) data of execution quality. The take-away is that the results of the pre-trade and post-trade analyses are in fact very similar. 2 Methodology Most existing methods for evaluating the quality of execution are based on analyses in which the reference price is based on the ‘Best Bid Best Offer’ (BBBO). However, comparing post-trade transaction prices with pre￾trade quotes is problematic for various reasons. One being: while transaction-data is readily available among NCAs, order-data isn’t necessarily so. Therefore, many NCAs (who might be especially interested in these types of analyses) might not be able to consistently apply a methodology based on order-data. Another reason would be that pre-trade quotes might not necessarily take into account any hidden liquidity. When the best bid in a lit market is EUR 10.- and the best offer is EUR 11, there is often still liquidity to be found at, for instance, EUR 10.20 (think sniping orders, that only appear once the order to be traded with appears in the order book8 ). Hence, comparing the transaction price to the BBBO in the lit orderbook is theoretically unsound. A more accurate methodological basis to analyze quality of executions would be to use a benchmark based on actual transactions: the reference price. Such a reference price based on actual prices could be constructed either through the concept of (I) an average price of the ten9 reference trading venues or (II) by establishing a ‘range’ of prices consisting of the “minimum” and “maximum” price on the ten reference trading venues. In this analysis the AFM used reference price II, which allows one to determine whether the price of a specific trade was worse, better or similar than the price on the reference trading venues. In using reference price II, we chose a more conservative approach. In the MiFID II-dataset (being the transaction reported to NCAs) we identify the “client-side” by either taking (1) the opposite side in a transaction with a market maker (if there is only one market maker active on the relevant market) or (2) taking the side of the “Natural Person” (where available). The method for
  identification of the client-side depends on the particular trading venue to which the model is applied. This step is crucial, since it allows us to qualify the difference with the reference price as being worse, better or similar for the client. 7 Annexes III was added in version 2 of this paper (March 2022). In possible future versions of this paper, additional analysis and annexes might be added. 8 Another issue has to do with iceberg-orders, which might show only a particular number of shares on offer, while there are in fact more shares on offer. 9 The AFM used 10 trading venues, yet this number could be adjusted.

6 Assessing the quality of executions on trading venues Basically, the Comparative Pricing Model works as follows:

• For every transaction on the relevant trading venue, we take the price of execution and whether the
  client-side is buyer or seller.
• We compare this price with the execution prices in the same instrument in the same second on ten
  other venues.
• Each execution on the relevant trading venue is labeled as better, similar, or worse. Assuming the
  client-side is the buyer in the transaction, the execution is considered better if the price is lower than the prices of all transactions in the same instrument in the same second on all other trading venues, similar if the price is between the lowest and highest price, and worse otherwise (vice versa for transactions where the client-side is a seller). The AFM only includes transactions in Dutch shares in its analyses. The reason being that the AFM – in principle – receives all transactions in all Dutch shares within the EEA – irrespective of the trading venue being traded on. Hence, we can be sure that for any Dutch share we have plenty of transactions on reference trading venues to compare transactions with. For this reason, we recommend any other NCA to apply the method on local shares, or at least on order flow which provides sufficient data points for a fair comparison. Besides labeling a transaction as being “Worse”, “Better” or “Similar” in execution, one can quantify how much worse or better a particular execution is when looking (solely) at the price component. For this – and our complete methodology – we refer the reader to Annex I.

7 Assessing the quality of executions on trading venues 3 Research outcomes The AFM applied the methodology to the transactions in Dutch shares on three trading venues (two PFOF trading venues and one non-PFOF trading venue) and to one low-cost investment firm. These venues and investment firm were selected because their activities and services are provided to retail clients across multiple EU member states and they reported a relatively large number of transactions (hence data points). The analyses found that the majority of retail client transactions on the two PFOF trading venues were executed at prices worse than transactions on the reference trading venues. On the non-PFOF trading venue, most of the retail client transactions have similar execution prices when compared to the reference trading venues. For the investment firm, the transaction prices are almost equally divided among the labels worse, better and similar. In the remainder of this chapter, we present the results for the three trading venues and the low-cost investment firm in detail. PFOF trading venue X The outcomes below show that retail clients got a worse price in 68-72% of the cases and a better price in 5- 8% of the transactions when compared to prices in the reference market(s). That is: 68.8% was worse when we compare the executions to executions on ten other trading venues and 72.0% was worse if we compare the transactions solely with transactions on Euronext Amsterdam (which is the most liquid trading venue available in our dataset). We found that – on average, when compared to executions on Euronext Amsterdam – clients trading on trading venue X are paying 4.80 basis points extra per transaction. For a transaction of € 3,000 this means the price is worse by € 1.44. Trading venue X is a PFOF trading venue operating with a regulated market license. Trading venue X has one market maker acting as the counterparty for nearly all retail client orders in shares. PFOF trading venue X Execution prices vs other trading venues Worse Similar Better Execution price vs other venues 68.8% 23.5% 7.6% Based on > 140,000 transactions Execution price vs Euronext Amsterdam 72.0% 22.4% 5.7% Based on 124,904 transactions Price improvement or deterioration Average price deterioration vs Euronext Amsterdam 4.8 bps Based on 124,904 transactions Average price deterioration for a trade of € 1,000 € 0.48 Based on 124,904 transactions Average price deterioration for a trade of € 3,000 € 1.44 Based on 124,904 transactions

8 Assessing the quality of executions on trading venues

9 Assessing the quality of executions on trading venues PFOF trading venue Y The outcomes below show that clients got a worse price in 81-83% of the cases and a better price in 6-7% of the transactions when compared to prices in the reference market(s). That is: 81.5% was worse when compared to transactions on ten other trading venues and 83.3% was worse when compared to transactions executed on Euronext Amsterdam. We found that – on average, when compared to executions on Euronext Amsterdam – clients trading on Trading venue Y are paying 11.5 basis points extra per transaction, or € 3.46 worse for a transaction of € 3,000. Trading venue Y is a PFOF trading venue operating with a regulated market license. Our data shows the trading venue seems to handle retail client orders from primarily one low-cost broker. Trading venue Y has one market maker acting as the counterparty for nearly all retail client orders in shares.

10 Assessing the quality of executions on trading venues PFOF trading venue Y Execution prices vs other trading venues Worse Similar Better Execution price vs other venues 81.5% 11.7% 6.8% Based on > 35,000 transactions Execution price vs Euronext Amsterdam 83.3% 9.8% 6.9% Based on 29,940 transactions Price improvement or deterioration Average price deterioration vs Euronext Amsterdam 11.5 bps Based on 29,940 transactions Average price deterioration for a trade of € 1,000 € 1.15 Based on 29,940 transactions Average price deterioration for a trade of € 3,000 € 3.46 Based on 29,940 transactions

11 Assessing the quality of executions on trading venues

12 Assessing the quality of executions on trading venues Trading venue Z The outcomes below show that clients got a worse price in 22-24% of the cases and a better price in 1-2% of the transactions when compared to prices in the reference market(s). That is: 22.1% was worse compared to transactions on ten other trading venues and 23.5% was worse compared to transactions executed on Euronext Amsterdam. We found that – on average, when compared to executions on Euronext Amsterdam – clients trading on Trading venue Z are paying 0.8 basis points extra per transaction, or € 0.24 worse on a transaction of € 3,000. Trading venue Z is operating with a regulated market license. The trading venue is used by all kinds of firms, ranging from low cost neo-brokers to more traditional banks and investment firms. Both low cost neo-brokers with and without an inducement business model use Trading venue Z for execution of their clients’ orders. The trading venue allows for multiple market makers to provide liquidity and act as counterparty for client orders. trading venue Z Execution prices vs other trading venues Worse Similar Better Execution price vs other venues 22.1% 76.2% 1.7% Based on > 160,000 transactions Execution price vs Euronext Amsterdam 23.5% 74.6% 1.9% Based on 141,461 transactions Price improvement or deterioration Average price deterioration vs Euronext Amsterdam 0.8 bps Based on 142,461 transactions Average price deterioration for a trade of € 1,000 € 0.08 Based on 142,461 transactions Average price deterioration for a trade of € 3,000 € 0.24 Based on 142,461 transactions

13 Assessing the quality of executions on trading venues

14 Assessing the quality of executions on trading venues

Investment firm A
For the investment firm, we slightly modified the methodology. Since the investment firm is not a trading venue, we compare transactions via the investment firm to transactions occurring via other investment firms. The other steps in the methodology remain the same. The outcome below shows that clients got a worse price in 31-34% of the cases and a better price in 32-36% of the transactions compared to prices via other investment firms (and via other investment firms on Euronext Amsterdam). That is: 33.5% was worse when compared to transactions via other investment firms and 30.8% was worse when compared to transactions executed via other investment firms on Euronext Amsterdam. We found that – on average, when compared to executions on Euronext Amsterdam – clients trading via Investment firm A are paying 1.38 basis points extra per transaction (or € 0.42 worse for transaction of € 3,000).

15 Assessing the quality of executions on trading venues Investment firm A Execution prices vs other investment firms and Euronext Amsterdam Worse Similar Better Execution price vs other investment firms 33.5% 34.4% 32.1% Based on > 100,000 transactions Execution price vs Euronext Amsterdam 30.8% 33.5% 35.7% Based on 84,977 transactions Price improvement or deterioration Average price deterioration vs Euronext Amsterdam 1.4 bps Based on 84,977 transactions Average price deterioration for a trade of € 1,000 € 0.14 Based on 84,977 transactions Average price deterioration for a trade of € 3,000 € 0.42 Based on 84,977 transactions

16 Assessing the quality of executions on trading venues

17 Assessing the quality of executions on trading venues
Annex I Specifications of the Comparative Pricing Model
1) Construct target dataset “A” by selecting from the consolidated TRS/MIFID II tape only transactions;
a) on the MIC of the trading venue of which the quality of execution is to be assessed (the “relevant market”) b) within a specific subset of time10 c) in a specific subset of financial instruments11 for which the Supervisor is Relevant Competent Authority12
2) Construct reference dataset “R” by selecting from the consolidated TRS/MIFID II tape only transactions;
a) within the specific subset of time as defined under 1b b) in the specific subset of financial instruments as defined under 1c c) on the MICs of the most liquid trading venues in that instrument13
3) Add a column to A that identifies the side of the client in each transaction14
.
4) Make a combined data table “C” where – for each transaction in A – we take (when available) the minimum￾and maximum price as traded in the relevant instrument in the relevant second in dataset R
15
. For example: 10-
11 would mean that trades in that instrument in that second on all MICs in dataset R, were executed between prices 10 and 11. N.B. Depending on the liquidity and the volatility of the instrument the timeframe could be made shorter or longer. E.a. when it concerns trading in short term government bonds, prices are relatively stable, hence even a day could still provide significant insights in execution costs.
5) Add a column to C that identifies deviations between the price paid on the relevant market (A) and prices paid
on other trading venues (R) in the same instrument in the same second. In case the transaction is a buy￾transaction for the client (as identified in step 3), then:
(1) assign the transaction a “Better execution” if the price of the transaction is lower than the minimum price paid in any transaction in the same instrument in the same second on another trading venue (as occurring in R) 10 The period of dates/time should be sufficiently large as to allow for a sufficient number of data-points. The prime consideration here is whether – after joining datasets A and R in step 4 – sufficient data-points remain as to allow for valid conclusions. In our analyses we used the period of 2021-01-01 until 2021-07-01. 11 The number of instruments should be sufficiently large as to allow for a sufficient number of data-points. 12 The analyses should only be done on financial instruments for which one is Relevant Competent Authority. The reason being that one receives (in principle) transactions done on all trading in the EEA, which allows for comparisons across trading venues. 13 Excluding the MIC of the relevant market, “XOFF” and “XXXX”. In our analyses we took 10 reference markets. 14 For single dealer trading venues, one can take the side opposite from the side of the dealer. For trading venues with multiple market makers, one can identify all natural persons as being retail-clients and/or focusing on executions by a particular broker, then take those clients as being the relevant side. 15 An “inner join” of A and R. We round datetime of transactions to the nearest second.

18 Assessing the quality of executions on trading venues (2) assign the transaction a “Worse execution” if the price of the transaction is higher than the maximum price paid in the same instrument in the same second on another trading venue (as occurring in R) (3) Otherwise assign a “Similar execution” Vice versa for sell-transactions.
6) Report the counts (e.a., number of “Worse execution”, “Better Execution” and “Similar execution”)
In addition, one could compute the deviation between the price of the relevant transactions and the average price of all transactions in the same instrument in the same second as occurring on the other trading venues. We convert this deviation to basis points, then take the average to obtain the difference in cost of execution on the relevant market as compared to the reference trading venues. Example report calculated best-ex indicators for platform X Reference market = Top 10 markets in relevant instruments in relevant period of time A = the cost of trading away from the average price on the reference market(s)16 B = the number of price improvements compared to the reference markets C = the number of price deteriorations compared to the reference markets D = the number of “similar” executions compared to the reference markets 16 To be calculated under 6).

19 Assessing the quality of executions on trading venues

TARGET DATASET REFERENCE DATASET
Quarter Q1 2021 Q1 2021
Number of transactions 318,053 1,376,872
Value of transactions EUR 25,997,354. - EUR 1,524,978,132

INDICATOR INDICATORS FOR PLATFORM X
A 0.1 % based on total euro value
B 3% based on total number of transactions
C 21% based on total number of transactions
D 76% based on total number of transactions

20 Assessing the quality of executions on trading venues
Annex II Questions and answers

1. Question: Could other National Competent Authorities also use the Comparative Pricing Model?
   Answer: Yes, they can. The model is applied to MIFID II-data and any National Competent Authority that receives such data can apply it. Furthermore, the AFM is happy to share the code used to do the analyses, so other NCAs can easily apply the method to their own datasets.
2. Question: Could trading venues or investment firms also use the Comparative Pricing Model?
   Answer: Trading venues or investment firms could combine data from their own client trades with the transaction publications mandated by MiFID II transparency, to arrive at similar reports.
3. Question: Why isn’t pre-trade data included in the analysis? And wouldn’t it be better to do so?
   Answer: One could argue that analyses based on pre-trade/order-data are also (or even more) accurate when it comes to assessing the quality of execution across trading venues. The reason being that pre-trade/order-data provides continuous prices, which means it could provide a reference price for the quality of execution at any point in time – not only at the points in time for which there is at least one matching transaction (in the same instrument) in the same second on another trading venue. The AFM has taken the approach based on post-trade data and discrete time for the following reasons:
   (1) While MIFID II-data is readily available for National Competent Authorities, pre-trade/order-trade isn’t necessarily so. Therefore, using transaction-data makes the model much easier to implement and apply by NCAs. (2) The continuous time data used in pre-trade based analyses can be affected by significant data quality issues related to latency and exact timestamps, where even a nanosecond can make a difference. Hence, methods solely using continuous time data might be more prone to error. (3) Also, transaction-data could be a better source for measuring the quality of execution than pre￾trade data since pre-trade data does not display hidden liquidity (e.g. iceberg-order and/or sniping orders). While pre-trade data surely is an indication for the likely price of execution of any transaction, it might therefore differ from the actual price paid or received by the client. We also refer to Annex III, in which we show that the results obtained from applying a pre-trade analysis are (very) similar to the results obtained via the Comparative Pricing Model.
4. Question: What timestamp is used to match transactions on the relevant market to the reference
   markets? And isn’t a period of one second too long to compare prices across trading venues? Answer: The AFM used the timestamp of each transaction rounded to the nearest second and compares transactions happening within that same second. The AFM realizes that prices could change (across trading venues) within that one second due to normal market dynamics. Hence a difference in the price of execution between trading venues in the same second doesn’t necessarily imply that quality of execution on one trading venue is better or worse – it could simply be due to uneven distribution of buying and selling orders. For example: one could have a client’s buy-transaction occurring at the start of a particular second (say “10:12:13:121”) being qualified as having “Worse Execution”, simply because the price on other trading venues moved down due to overall selling activity in the latter part of the same second (say “10:12:13:931”). However, using the same logic there will also be cases in which the transaction is qualified as “Better Execution” – even though this isn’t the case either.

21 Assessing the quality of executions on trading venues By including a sufficiently large number of data-points, we assume the overall results are – despite errors in individual cases – a reasonably good reflection of the actual quality of execution on the relevant trading venue.
5. Question: Could the Comparative Pricing Model also be used to assess the quality of execution in other
types of financial instruments than shares?
Answer: In principle the same method could be applied to many other asset classes. However, one (most likely) would have to make some modifications. One should for example consider the liquidity of the relevant financial instrument: an illiquid instrument might have none/few transactions occurring within the same second on another trading venue, forcing one to possibly extend the period beyond one second, for example.
6. Question: Why don’t you include volume in your analysis? Surely volume is relevant when comparing
the quality of execution of transactions?
Answer: The AFM realizes that comparing transactions with small volumes to transactions with large volumes could distort the results. After all: due to price-impact one might expect larger transactions to have (with regards to price) a worse execution than smaller transactions would have had. For the purposes of this paper, the AFM assumes the distribution of volume per transaction to be relatively similar across trading venues. By including a sufficiently large number of data-points, we assume the overall results are – despite errors in individual cases –a reasonably good reflection of the actual quality of execution on the relevant trading venue (according to the logic similar to the Answer to Q.4 “What timestamp is used to match transactions on the relevant venue to the reference markets? And isn’t a period of one second too long to compare prices across trading venues?”). The Comparative Pricing Model could – for future purposes – be extended as to correct for difference in volume per transactions. One could – for example – compare the quality of execution on a trading venue for different volumes per transaction (“Bottom 10%, “Median”, “Top 10%”).
7. Question: Why don’t you look at different order types (market-orders versus limit-orders)? What are
implications of not doing so?
Answer: The AFM doesn’t have data on order type available for each trading venue. Yet the AFM realizes that order types can play a role in determining quality of execution. For example: a buy￾transaction (entered via a “market order”) on the relevant market (which is matched to best offer of e.a. “10.72”) might be compared to transactions executed via limit orders occurring on other trading venues (“10.70”). Strictly speaking, this first transaction shouldn’t be qualified as having “Worse Execution”. However, according to the same reasoning, there will also be cases in which the transaction is qualified as “Better Execution”, even though this isn’t the case. By including a sufficiently large number of data￾points, we assume the overall results are – despite errors in individual cases – a reasonably good reflection of the actual quality of execution on the relevant trading venue. Still, the method could be refined by distinguishing between order-types.
8. Question: What is the impact of Data Quality on the output of your analysis?
Answer: Data Quality can have a large impact on any analyses. Since MIFID II-data is data reported by trading venues and firms, there could (in principle) be reporting issues. Therefore, we recommend always doing sanity-checks on at least the most important variables in your dataset. The most apparent example being the “timestamp” of a transaction, as it is crucial that trading venues and firms use synchronised clocks for different time-zones when reporting their transactions. Otherwise, one might be comparing transactions that were not actually executed within the same second.

22 Assessing the quality of executions on trading venues
9. Question: What does the analysis conclude about the practice of payment for order flow
Answer: The objective of the analyses is not to draw definitive conclusions about PFOF or the effect of PFOF on the quality of execution: the objective is to assess the execution quality with regards to the execution price – for both PFOF and non-PFOF trading venues. According to our analyses, the studied PFOF-trading venues offer worse execution prices than the reference trading venues. Yet there might be other factors (partially) responsible for the worse execution prices, such as lack of competition for the orders on the particular trading venue.

23 Assessing the quality of executions on trading venues
Annex III The Comparative Pricing Model and order-data analysis
In this paper, the AFM explains how a National Competent Authority (“NCA”) could go about using reported transaction-data to determine the quality of executions on a trading venue. In short: by comparing the prices of many transactions on the relevant trading venue with prices of transactions occurring at the same point in time on other trading venues, we established a robust approximation of the execution quality on the relevant trading venue. In response to our paper, the AFM noticed that some parties would have liked us to compare transaction-data with order-data (instead of with transaction-data). Especially: comparing the execution on the relevant trading venue with the best bid and best offer on a liquid reference market. Some respondents claimed such a method might provide different results or a possibly better method to assess execution quality. The AFM would like to refer to the chapters 1 Introduction and 2 Methodology for an explanation as to why we prefer the Comparative Pricing Model to an order-data based analysis. Since the AFM has order-data of Euronext Amsterdam at its disposal, we can indeed compare executions on the relevant trading venue with the relevant best bid and best offer at the time of the transaction on a liquid reference market. The most liquid reference market for Dutch shares is Euronext Amsterdam. As the AFM has this order-data at its disposal, we can compare the best bid and best offer on Euronext Amsterdam with the actual execution price of the relevant transactions on a millisecond level. Doing so, the AFM obtained the following results. The execution prices on the relevant PFOF venues were structurally worse compared to the quoted (best bid and best offer) prices on Euronext Amsterdam. The results are very similar to the ones obtained via the Comparative Pricing Model. We believe these results validate the Comparative Pricing Model and might show NCAs – especially those that don’t have order-data at their disposal – that the transaction-based analysis is indeed a valid method to use instead. Methodology used for the order-data based analysis Transaction-data (1) Select the trading venue to assess (2) Take 5 most traded Dutch shares on (1) in 202117 (3) Take a relevant time period (we took the first four months of 2021) which provides sufficient data points (4) Take all MiFID-II reported transactions on (1) in shares (2) in period (3) during regular market hours (5) Determine relevant side in transactions (4) (“Buy” or “Sell”) according to logic from the Comparative Pricing Model Order-data (6) Using Euronext Amsterdam order-data, take the best bid and best offer per millisecond in the shares (2) in period (3). Specifically: we only list the milliseconds – and best bids/best offer – in which the best bid or best offer changed. This is equivalent to saying the best bid and best offer was constant in between two subsequent timestamps. (7) For each transaction obtained via (5) (reported up to milliseconds) we find
the last change in best bid and best offer before the relevant transaction. This is equivalent to taking the best bid and best offer at Euronext Amsterdam at the millisecond of the transaction. 17 We use 5 shares because including more shares would be too computationally intensive.

24 Assessing the quality of executions on trading venues (8) We compare the execution following the same logic of the Comparative Pricing Model (replacing “maxprice” with “best offer” and “minprice” with “best bid”). For example: a buy-transaction would have “Better Execution” on the relevant trading venue if the price was lower than “best bid” at time of transaction; “Worse” in case price was higher than “best offer”; “Similar” otherwise. Doing so, we obtain comparisons such as the following:
And graphically:
Below, the results of these order-data analyses are included for PFOF trading venues X and Y and trading venue Z. These are the same trading venues as used in Chapter III of this paper. The results are similar to the Comparative Pricing Model. For instance, on PFOF trading venue X the Comparative Pricing Model found 72% worse executions in the same time period. This is 72.7% compared to Euronext in the order-data analysis. For PFOF trading venue Y this is 83.3% vs 77.9% and for trading venue Z 23.5% vs 26.9%. The second chart per trading venue shows the price improvements or price deteriorations versus the mid-price at Euronext Amsterdam.

25 Assessing the quality of executions on trading venues PFOF Trading Venue X

26 Assessing the quality of executions on trading venues PFOF Trading Venue Y

27 Assessing the quality of executions on trading venues Trading Venue Z

28 Assessing the quality of executions on trading venues Follow us: → The AFM is committed to promoting fair and transparent financial markets. As an independent market conduct authority, we contribute to a sustainable financial system and prosperity in the Netherlands. The text of this publication has been compiled with care and is informative in nature. No rights may be derived from it. Changes to national and international legislation and regulation may mean that the text is no longer fully up to date when you read it. The Dutch Authority for the Financial Markets is not liable for any consequences - such as losses incurred or lost profits - of any actions taken in connection with this text © Copyright AFM 2022 The Dutch Authority for the Financial Markets PO Box 11723 | 1001 GS Amsterdam Telephone +31 20 797 2000 www.afm.nl Data classification AFM - Public