| Message ID | 20210904064932.307610-1-kjain@linux.ibm.com (mailing list archive) |
|---|---|
| State | Changes Requested |
| Headers | show |
| Series | [1/3] perf: Add macros to specify onchip L2/L3 accesses | expand |
| Related | show |
Kajol Jain <kjain@linux.ibm.com> writes: > Add couple of new macros to represent onchip L2 and onchip L3 accesses. It would be "on chip". But I think this needs much more explanation, this is a generic header so these definitions need to make sense, and have an understood meaning, across all architectures. I think most people are going to read "on chip" as differentiating between an L2/L3 that is "on chip" vs "off chip". But the case you're trying to express is "another core's L2/L3 on the same chip as the CPU", vs "the current CPU's L2/L3". > diff --git a/include/uapi/linux/perf_event.h b/include/uapi/linux/perf_event.h > index f92880a15645..030b3e990ac3 100644 > --- a/include/uapi/linux/perf_event.h > +++ b/include/uapi/linux/perf_event.h > @@ -1265,7 +1265,9 @@ union perf_mem_data_src { > #define PERF_MEM_LVLNUM_L2 0x02 /* L2 */ > #define PERF_MEM_LVLNUM_L3 0x03 /* L3 */ > #define PERF_MEM_LVLNUM_L4 0x04 /* L4 */ > -/* 5-0xa available */ > +#define PERF_MEM_LVLNUM_OC_L2 0x05 /* On Chip L2 */ > +#define PERF_MEM_LVLNUM_OC_L3 0x06 /* On Chip L3 */ The obvious use for 5 is for "L5" and so on. I'm not sure adding new levels is the best idea, because these don't fit neatly into the hierarchy, they are off to the side. I wonder if we should use the remote field. ie. for another core's L2 we set: mem_lvl = PERF_MEM_LVL_L2 mem_remote = 1 Which would mean "remote L2", but not remote enough to be lvl = PERF_MEM_LVL_REM_CCE1. It would be printed by the existing tools/perf code as "Remote L2", vs "Remote cache (1 hop)", which seems OK. ie. we'd be able to express: Current core's L2: LVL_L2 Other core's L2: LVL_L2 | REMOTE Other chip's L2: LVL_REM_CCE1 | REMOTE And similarly for L3. I think that makes sense? Unless people think remote should be reserved to mean on another chip, though we already have REM_CCE1 for that. cheers
On Wed, Sep 08, 2021 at 05:17:53PM +1000, Michael Ellerman wrote: > Kajol Jain <kjain@linux.ibm.com> writes: > > diff --git a/include/uapi/linux/perf_event.h b/include/uapi/linux/perf_event.h > > index f92880a15645..030b3e990ac3 100644 > > --- a/include/uapi/linux/perf_event.h > > +++ b/include/uapi/linux/perf_event.h > > @@ -1265,7 +1265,9 @@ union perf_mem_data_src { > > #define PERF_MEM_LVLNUM_L2 0x02 /* L2 */ > > #define PERF_MEM_LVLNUM_L3 0x03 /* L3 */ > > #define PERF_MEM_LVLNUM_L4 0x04 /* L4 */ > > -/* 5-0xa available */ > > +#define PERF_MEM_LVLNUM_OC_L2 0x05 /* On Chip L2 */ > > +#define PERF_MEM_LVLNUM_OC_L3 0x06 /* On Chip L3 */ > > The obvious use for 5 is for "L5" and so on. > > I'm not sure adding new levels is the best idea, because these don't fit > neatly into the hierarchy, they are off to the side. > > > I wonder if we should use the remote field. > > ie. for another core's L2 we set: > > mem_lvl = PERF_MEM_LVL_L2 > mem_remote = 1 This mixes APIs (see below), IIUC the correct usage would be something like: lvl_num=L2 remote=1 > Which would mean "remote L2", but not remote enough to be > lvl = PERF_MEM_LVL_REM_CCE1. > > It would be printed by the existing tools/perf code as "Remote L2", vs > "Remote cache (1 hop)", which seems OK. > > > ie. we'd be able to express: > > Current core's L2: LVL_L2 > Other core's L2: LVL_L2 | REMOTE > Other chip's L2: LVL_REM_CCE1 | REMOTE > > And similarly for L3. > > I think that makes sense? Unless people think remote should be reserved > to mean on another chip, though we already have REM_CCE1 for that. IIRC the PERF_MEM_LVL_* namespace is somewhat depricated in favour of the newer composite PERF_MEM_{LVLNUM_,REMOTE_,SNOOPX_} fields. Of course, ABIs being what they are, we get to support both :/ But I'm not sure mixing them is a great idea. Also, clearly this could use a comment... The 'new' composite doesnt have a hops field because the hardware that nessecitated that change doesn't report it, but we could easily add a field there. Suppose we add, mem_hops:3 (would 6 hops be too small?) and the corresponding PERF_MEM_HOPS_{NA, 0..6} Then I suppose you can encode things like: L2 - local L2 L2 | REMOTE - remote L2 at an unspecified distance (NA) L2 | REMOTE | HOPS_0 - remote L2 on the same node L2 | REMOTE | HOPS_1 - remote L2 on a node 1 removed Would that work?
Peter Zijlstra <peterz@infradead.org> writes: > On Wed, Sep 08, 2021 at 05:17:53PM +1000, Michael Ellerman wrote: >> Kajol Jain <kjain@linux.ibm.com> writes: > >> > diff --git a/include/uapi/linux/perf_event.h b/include/uapi/linux/perf_event.h >> > index f92880a15645..030b3e990ac3 100644 >> > --- a/include/uapi/linux/perf_event.h >> > +++ b/include/uapi/linux/perf_event.h >> > @@ -1265,7 +1265,9 @@ union perf_mem_data_src { >> > #define PERF_MEM_LVLNUM_L2 0x02 /* L2 */ >> > #define PERF_MEM_LVLNUM_L3 0x03 /* L3 */ >> > #define PERF_MEM_LVLNUM_L4 0x04 /* L4 */ >> > -/* 5-0xa available */ >> > +#define PERF_MEM_LVLNUM_OC_L2 0x05 /* On Chip L2 */ >> > +#define PERF_MEM_LVLNUM_OC_L3 0x06 /* On Chip L3 */ >> >> The obvious use for 5 is for "L5" and so on. >> >> I'm not sure adding new levels is the best idea, because these don't fit >> neatly into the hierarchy, they are off to the side. >> >> >> I wonder if we should use the remote field. >> >> ie. for another core's L2 we set: >> >> mem_lvl = PERF_MEM_LVL_L2 >> mem_remote = 1 > > This mixes APIs (see below), IIUC the correct usage would be something > like: lvl_num=L2 remote=1 Aha, I was wondering how lvl and lvl_num were supposed to interact. >> Which would mean "remote L2", but not remote enough to be >> lvl = PERF_MEM_LVL_REM_CCE1. >> >> It would be printed by the existing tools/perf code as "Remote L2", vs >> "Remote cache (1 hop)", which seems OK. >> >> >> ie. we'd be able to express: >> >> Current core's L2: LVL_L2 >> Other core's L2: LVL_L2 | REMOTE >> Other chip's L2: LVL_REM_CCE1 | REMOTE >> >> And similarly for L3. >> >> I think that makes sense? Unless people think remote should be reserved >> to mean on another chip, though we already have REM_CCE1 for that. > > IIRC the PERF_MEM_LVL_* namespace is somewhat depricated in favour of > the newer composite PERF_MEM_{LVLNUM_,REMOTE_,SNOOPX_} fields. Of > course, ABIs being what they are, we get to support both :/ But I'm not > sure mixing them is a great idea. OK. > Also, clearly this could use a comment... > > The 'new' composite doesnt have a hops field because the hardware that > nessecitated that change doesn't report it, but we could easily add a > field there. > > Suppose we add, mem_hops:3 (would 6 hops be too small?) and the > corresponding PERF_MEM_HOPS_{NA, 0..6} It's really 7 if we use remote && hop = 0 to mean the first hop. If we're wanting to use some of the hop levels to represent intra-chip/package hops then we could possibly use them all on a really big system. eg. you could imagine something like: L2 | - local L2 L2 | REMOTE | HOPS_0 - L2 of neighbour core L2 | REMOTE | HOPS_1 - L2 of near core on same chip (same 1/2 of chip) L2 | REMOTE | HOPS_2 - L2 of far core on same chip (other 1/2 of chip) L2 | REMOTE | HOPS_3 - L2 of sibling chip in same package L2 | REMOTE | HOPS_4 - L2 on separate package 1 hop away L2 | REMOTE | HOPS_5 - L2 on separate package 2 hops away L2 | REMOTE | HOPS_6 - L2 on separate package 3 hops away Whether it's useful to represent all those levels I'm not sure, but it's probably good if we have the ability. I guess I'm 50/50 on whether that's enough levels, or whether we want another bit to allow for future growth. > Then I suppose you can encode things like: > > L2 - local L2 > L2 | REMOTE - remote L2 at an unspecified distance (NA) > L2 | REMOTE | HOPS_0 - remote L2 on the same node > L2 | REMOTE | HOPS_1 - remote L2 on a node 1 removed > > Would that work? Yeah that looks good to me. cheers
On Thu, Sep 09, 2021 at 10:45:54PM +1000, Michael Ellerman wrote: > > The 'new' composite doesnt have a hops field because the hardware that > > nessecitated that change doesn't report it, but we could easily add a > > field there. > > > > Suppose we add, mem_hops:3 (would 6 hops be too small?) and the > > corresponding PERF_MEM_HOPS_{NA, 0..6} > > It's really 7 if we use remote && hop = 0 to mean the first hop. I don't think we can do that, becaus of backward compat. Currently: lvl_num=DRAM, remote=1 denites: "Remote DRAM of any distance". Effectively it would have the new hops field filled with zeros though, so if you then decode with the hops field added it suddenly becomes: lvl_num=DRAM, remote=1, hops=0 and reads like: "Remote DRAM of 0 hops" which is quite daft. Therefore 0 really must denote a 'N/A'. > If we're wanting to use some of the hop levels to represent > intra-chip/package hops then we could possibly use them all on a really > big system. > > eg. you could imagine something like: > > L2 | - local L2 > L2 | REMOTE | HOPS_0 - L2 of neighbour core > L2 | REMOTE | HOPS_1 - L2 of near core on same chip (same 1/2 of chip) > L2 | REMOTE | HOPS_2 - L2 of far core on same chip (other 1/2 of chip) > L2 | REMOTE | HOPS_3 - L2 of sibling chip in same package > L2 | REMOTE | HOPS_4 - L2 on separate package 1 hop away > L2 | REMOTE | HOPS_5 - L2 on separate package 2 hops away > L2 | REMOTE | HOPS_6 - L2 on separate package 3 hops away > > > Whether it's useful to represent all those levels I'm not sure, but it's > probably good if we have the ability. I'm thinking we ought to keep hops as steps along the NUMA fabric, with 0 hops being the local node. That only gets us: L2, remote=0, hops=HOPS_0 -- our L2 L2, remote=1, hops=HOPS_0 -- L2 on the local node but not ours L2, remote=1, hops!=HOPS_0 -- L2 on a remote node > I guess I'm 50/50 on whether that's enough levels, or whether we want > another bit to allow for future growth. Right, possibly safer to add one extra bit while we can.... I suppose.
Peter Zijlstra <peterz@infradead.org> writes: > On Thu, Sep 09, 2021 at 10:45:54PM +1000, Michael Ellerman wrote: > >> > The 'new' composite doesnt have a hops field because the hardware that >> > nessecitated that change doesn't report it, but we could easily add a >> > field there. >> > >> > Suppose we add, mem_hops:3 (would 6 hops be too small?) and the >> > corresponding PERF_MEM_HOPS_{NA, 0..6} >> >> It's really 7 if we use remote && hop = 0 to mean the first hop. > > I don't think we can do that, becaus of backward compat. Currently: > > lvl_num=DRAM, remote=1 > > denites: "Remote DRAM of any distance". Effectively it would have the new > hops field filled with zeros though, so if you then decode with the hops > field added it suddenly becomes: > > lvl_num=DRAM, remote=1, hops=0 > > and reads like: "Remote DRAM of 0 hops" which is quite daft. Therefore 0 > really must denote a 'N/A'. Ah yeah, duh, it needs to be backward compatible. >> If we're wanting to use some of the hop levels to represent >> intra-chip/package hops then we could possibly use them all on a really >> big system. >> >> eg. you could imagine something like: >> >> L2 | - local L2 >> L2 | REMOTE | HOPS_0 - L2 of neighbour core >> L2 | REMOTE | HOPS_1 - L2 of near core on same chip (same 1/2 of chip) >> L2 | REMOTE | HOPS_2 - L2 of far core on same chip (other 1/2 of chip) >> L2 | REMOTE | HOPS_3 - L2 of sibling chip in same package >> L2 | REMOTE | HOPS_4 - L2 on separate package 1 hop away >> L2 | REMOTE | HOPS_5 - L2 on separate package 2 hops away >> L2 | REMOTE | HOPS_6 - L2 on separate package 3 hops away >> >> >> Whether it's useful to represent all those levels I'm not sure, but it's >> probably good if we have the ability. > > I'm thinking we ought to keep hops as steps along the NUMA fabric, with > 0 hops being the local node. That only gets us: > > L2, remote=0, hops=HOPS_0 -- our L2 > L2, remote=1, hops=HOPS_0 -- L2 on the local node but not ours > L2, remote=1, hops!=HOPS_0 -- L2 on a remote node Hmm. I'm not sure about tying it directly to NUMA hops. I worry we're going to see more and more systems where there's a hierarchy within the chip/package, in addition to the traditional NUMA hierarchy. Although then I guess it becomes a question of what exactly is a NUMA hop, maybe the answer is that on those future systems those intra-chip/package hops should be represented as NUMA hops. It's not like we have a hard definition of what a NUMA hop is? >> I guess I'm 50/50 on whether that's enough levels, or whether we want >> another bit to allow for future growth. > > Right, possibly safer to add one extra bit while we can.... I suppose. Equally it's not _that_ hard to add another bit later (if there's still one free), makes the API a little uglier to use, but not the end of the world. cheers
On Tue, Sep 14, 2021 at 08:40:38PM +1000, Michael Ellerman wrote: > Peter Zijlstra <peterz@infradead.org> writes: > > I'm thinking we ought to keep hops as steps along the NUMA fabric, with > > 0 hops being the local node. That only gets us: > > > > L2, remote=0, hops=HOPS_0 -- our L2 > > L2, remote=1, hops=HOPS_0 -- L2 on the local node but not ours > > L2, remote=1, hops!=HOPS_0 -- L2 on a remote node > > Hmm. I'm not sure about tying it directly to NUMA hops. I worry we're > going to see more and more systems where there's a hierarchy within the > chip/package, in addition to the traditional NUMA hierarchy. > > Although then I guess it becomes a question of what exactly is a NUMA > hop, maybe the answer is that on those future systems those > intra-chip/package hops should be represented as NUMA hops. > > It's not like we have a hard definition of what a NUMA hop is? Not really, typically whatever the BIOS/DT/whatever tables tell us. I think in case of Power you're mostly making things up in software :-) But yeah, I think we have plenty wriggle room there.
Peter Zijlstra <peterz@infradead.org> writes: > On Tue, Sep 14, 2021 at 08:40:38PM +1000, Michael Ellerman wrote: >> Peter Zijlstra <peterz@infradead.org> writes: > >> > I'm thinking we ought to keep hops as steps along the NUMA fabric, with >> > 0 hops being the local node. That only gets us: >> > >> > L2, remote=0, hops=HOPS_0 -- our L2 >> > L2, remote=1, hops=HOPS_0 -- L2 on the local node but not ours >> > L2, remote=1, hops!=HOPS_0 -- L2 on a remote node >> >> Hmm. I'm not sure about tying it directly to NUMA hops. I worry we're >> going to see more and more systems where there's a hierarchy within the >> chip/package, in addition to the traditional NUMA hierarchy. >> >> Although then I guess it becomes a question of what exactly is a NUMA >> hop, maybe the answer is that on those future systems those >> intra-chip/package hops should be represented as NUMA hops. >> >> It's not like we have a hard definition of what a NUMA hop is? > > Not really, typically whatever the BIOS/DT/whatever tables tell us. I > think in case of Power you're mostly making things up in software :-) Firmware is software so yes :) > But yeah, I think we have plenty wriggle room there. OK. cheers
diff --git a/include/uapi/linux/perf_event.h b/include/uapi/linux/perf_event.h index f92880a15645..030b3e990ac3 100644 --- a/include/uapi/linux/perf_event.h +++ b/include/uapi/linux/perf_event.h @@ -1265,7 +1265,9 @@ union perf_mem_data_src { #define PERF_MEM_LVLNUM_L2 0x02 /* L2 */ #define PERF_MEM_LVLNUM_L3 0x03 /* L3 */ #define PERF_MEM_LVLNUM_L4 0x04 /* L4 */ -/* 5-0xa available */ +#define PERF_MEM_LVLNUM_OC_L2 0x05 /* On Chip L2 */ +#define PERF_MEM_LVLNUM_OC_L3 0x06 /* On Chip L3 */ +/* 7-0xa available */ #define PERF_MEM_LVLNUM_ANY_CACHE 0x0b /* Any cache */ #define PERF_MEM_LVLNUM_LFB 0x0c /* LFB */ #define PERF_MEM_LVLNUM_RAM 0x0d /* RAM */
Add couple of new macros to represent onchip L2 and onchip L3 accesses. Signed-off-by: Kajol Jain <kjain@linux.ibm.com> --- include/uapi/linux/perf_event.h | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)