| Message ID | 3bb44d92-b2ff-e197-4bdf-ec6d588d6dab@linux.vnet.ibm.com (mailing list archive) |
|---|---|
| State | Superseded, archived |
| Headers | show |
On Tue, May 23, 2017 at 10:15:44AM -0500, Michael Bringmann wrote: >+static void setup_nodes(void) >+{ >+ int i, l = 32 /* MAX_NUMNODES */; >+ >+ for (i = 0; i < l; i++) { >+ if (!node_possible(i)) { >+ setup_node_data(i, 0, 0); >+ node_set(i, node_possible_map); >+ } >+ } >+} This seems to be a workaround for 3af229f2071f ("powerpc/numa: Reset node_possible_map to only node_online_map"). Balbir, you have a patchset which reverts it. Do you think that will be getting merged? http://lkml.kernel.org/r/1479253501-26261-1-git-send-email-bsingharora@gmail.com (see patch 3/3)
On 05/23/2017 10:52 AM, Reza Arbab wrote: > On Tue, May 23, 2017 at 10:15:44AM -0500, Michael Bringmann wrote: >> +static void setup_nodes(void) >> +{ >> + int i, l = 32 /* MAX_NUMNODES */; >> + >> + for (i = 0; i < l; i++) { >> + if (!node_possible(i)) { >> + setup_node_data(i, 0, 0); >> + node_set(i, node_possible_map); >> + } >> + } >> +} > > This seems to be a workaround for 3af229f2071f ("powerpc/numa: Reset node_possible_map to only node_online_map"). They may be related, but that commit is not a replacement. The above patch ensures that there are enough of the nodes initialized at startup to allow for memory hot-add into a node that was not used at boot. (See 'setup_node_data' function in 'numa.c'.) That and recording that the node was initialized. I didn't see where any part of commit 3af229f2071f would touch the 'node_possible_map' which is needed by 'numa.c' and 'workqueue.c'. The nodemask created and updated by 'mem_cgroup_may_update_nodemask()' does not appear to be the same mask. > > Balbir, you have a patchset which reverts it. Do you think that will be getting merged? > > http://lkml.kernel.org/r/1479253501-26261-1-git-send-email-bsingharora@gmail.com > (see patch 3/3) >
On Tue, May 23, 2017 at 03:05:08PM -0500, Michael Bringmann wrote: >On 05/23/2017 10:52 AM, Reza Arbab wrote: >> On Tue, May 23, 2017 at 10:15:44AM -0500, Michael Bringmann wrote: >>> +static void setup_nodes(void) >>> +{ >>> + int i, l = 32 /* MAX_NUMNODES */; >>> + >>> + for (i = 0; i < l; i++) { >>> + if (!node_possible(i)) { >>> + setup_node_data(i, 0, 0); >>> + node_set(i, node_possible_map); >>> + } >>> + } >>> +} >> >> This seems to be a workaround for 3af229f2071f ("powerpc/numa: Reset node_possible_map to only node_online_map"). > >They may be related, but that commit is not a replacement. The above patch ensures that >there are enough of the nodes initialized at startup to allow for memory hot-add into a >node that was not used at boot. (See 'setup_node_data' function in 'numa.c'.) That and >recording that the node was initialized. Is it really necessary to preinitialize these empty nodes using setup_node_data()? When you do memory hotadd into a node that was not used at boot, the node data already gets set up by add_memory add_memory_resource hotadd_new_pgdat arch_alloc_nodedata <-- allocs the pg_data_t ... free_area_init_node <-- sets NODE_DATA(nid)->node_id, etc. Removing setup_node_data() from that loop leaves only the call to node_set(). If 3af229f2071f (which reduces node_possible_map) was reverted, you wouldn't need to do that either. >I didn't see where any part of commit 3af229f2071f would touch the 'node_possible_map' >which is needed by 'numa.c' and 'workqueue.c'. The nodemask created and updated by >'mem_cgroup_may_update_nodemask()' does not appear to be the same mask. Are you sure you're looking at 3af229f2071f? It only adds one line of code; the reduction of node_possible_map.
On 05/23/2017 04:49 PM, Reza Arbab wrote: > On Tue, May 23, 2017 at 03:05:08PM -0500, Michael Bringmann wrote: >> On 05/23/2017 10:52 AM, Reza Arbab wrote: >>> On Tue, May 23, 2017 at 10:15:44AM -0500, Michael Bringmann wrote: >>>> +static void setup_nodes(void) >>>> +{ >>>> + int i, l = 32 /* MAX_NUMNODES */; >>>> + >>>> + for (i = 0; i < l; i++) { >>>> + if (!node_possible(i)) { >>>> + setup_node_data(i, 0, 0); >>>> + node_set(i, node_possible_map); >>>> + } >>>> + } >>>> +} >>> >>> This seems to be a workaround for 3af229f2071f ("powerpc/numa: Reset node_possible_map to only node_online_map"). >> >> They may be related, but that commit is not a replacement. The above patch ensures that >> there are enough of the nodes initialized at startup to allow for memory hot-add into a >> node that was not used at boot. (See 'setup_node_data' function in 'numa.c'.) That and >> recording that the node was initialized. > > Is it really necessary to preinitialize these empty nodes using setup_node_data()? When you do memory hotadd into a node that was not used at boot, the node data already gets set up by > > add_memory > add_memory_resource > hotadd_new_pgdat > arch_alloc_nodedata <-- allocs the pg_data_t > ... > free_area_init_node <-- sets NODE_DATA(nid)->node_id, etc. I see that code now, but for some reason it did not work when I hot-added memory. > > Removing setup_node_data() from that loop leaves only the call to node_set(). If 3af229f2071f (which reduces node_possible_map) was reverted, you wouldn't need to do that either. > >> I didn't see where any part of commit 3af229f2071f would touch the 'node_possible_map' >> which is needed by 'numa.c' and 'workqueue.c'. The nodemask created and updated by >> 'mem_cgroup_may_update_nodemask()' does not appear to be the same mask. > > Are you sure you're looking at 3af229f2071f? It only adds one line of code; the reduction of node_possible_map. > The 3rd file in the patch set removes, - nodes_and(node_possible_map, node_possible_map, node_online_map); I need to add bits to 'node_possible_map' -- bits which may not be used for the memory at boot, but which would be used when memory is hot-added later. I haven't found anything outside of the boot code that adds bits to the 'possible' mask.
On 05/23/2017 04:49 PM, Reza Arbab wrote: > On Tue, May 23, 2017 at 03:05:08PM -0500, Michael Bringmann wrote: >> On 05/23/2017 10:52 AM, Reza Arbab wrote: >>> On Tue, May 23, 2017 at 10:15:44AM -0500, Michael Bringmann wrote: >>>> +static void setup_nodes(void) >>>> +{ >>>> + int i, l = 32 /* MAX_NUMNODES */; >>>> + >>>> + for (i = 0; i < l; i++) { >>>> + if (!node_possible(i)) { >>>> + setup_node_data(i, 0, 0); >>>> + node_set(i, node_possible_map); >>>> + } >>>> + } >>>> +} >>> >>> This seems to be a workaround for 3af229f2071f ("powerpc/numa: Reset node_possible_map to only node_online_map"). >> >> They may be related, but that commit is not a replacement. The above patch ensures that >> there are enough of the nodes initialized at startup to allow for memory hot-add into a >> node that was not used at boot. (See 'setup_node_data' function in 'numa.c'.) That and >> recording that the node was initialized. > > Is it really necessary to preinitialize these empty nodes using setup_node_data()? When you do memory hotadd into a node that was not used at boot, the node data already gets set up by > > add_memory > add_memory_resource > hotadd_new_pgdat > arch_alloc_nodedata <-- allocs the pg_data_t > ... > free_area_init_node <-- sets NODE_DATA(nid)->node_id, etc. > > Removing setup_node_data() from that loop leaves only the call to node_set(). If 3af229f2071f (which reduces node_possible_map) was reverted, you wouldn't need to do that either. With or without 3af229f2071f, we would still need to add something, somewhere to add new bits to the 'node_possible_map'. That is not being done. > >> I didn't see where any part of commit 3af229f2071f would touch the 'node_possible_map' >> which is needed by 'numa.c' and 'workqueue.c'. The nodemask created and updated by >> 'mem_cgroup_may_update_nodemask()' does not appear to be the same mask. > > Are you sure you're looking at 3af229f2071f? It only adds one line of code; the reduction of node_possible_map. >
Michael Bringmann <mwb@linux.vnet.ibm.com> writes: > On 05/23/2017 04:49 PM, Reza Arbab wrote: >> On Tue, May 23, 2017 at 03:05:08PM -0500, Michael Bringmann wrote: >>> On 05/23/2017 10:52 AM, Reza Arbab wrote: >>>> On Tue, May 23, 2017 at 10:15:44AM -0500, Michael Bringmann wrote: >>>>> +static void setup_nodes(void) >>>>> +{ >>>>> + int i, l = 32 /* MAX_NUMNODES */; >>>>> + >>>>> + for (i = 0; i < l; i++) { >>>>> + if (!node_possible(i)) { >>>>> + setup_node_data(i, 0, 0); >>>>> + node_set(i, node_possible_map); >>>>> + } >>>>> + } >>>>> +} >>>> >>>> This seems to be a workaround for 3af229f2071f ("powerpc/numa: Reset node_possible_map to only node_online_map"). >>> >>> They may be related, but that commit is not a replacement. The above patch ensures that >>> there are enough of the nodes initialized at startup to allow for memory hot-add into a >>> node that was not used at boot. (See 'setup_node_data' function in 'numa.c'.) That and >>> recording that the node was initialized. >> >> Is it really necessary to preinitialize these empty nodes using setup_node_data()? When you do memory hotadd into a node that was not used at boot, the node data already gets set up by >> >> add_memory >> add_memory_resource >> hotadd_new_pgdat >> arch_alloc_nodedata <-- allocs the pg_data_t >> ... >> free_area_init_node <-- sets NODE_DATA(nid)->node_id, etc. >> >> Removing setup_node_data() from that loop leaves only the call to node_set(). If 3af229f2071f (which reduces node_possible_map) was reverted, you wouldn't need to do that either. > > With or without 3af229f2071f, we would still need to add something, somewhere to add new > bits to the 'node_possible_map'. That is not being done. You mustn't add bits to the possible map after boot. That's its purpose, to tell you what nodes could ever *possibly* exist. cheers
On Tue, May 23, 2017 at 05:44:23PM -0500, Michael Bringmann wrote: >On 05/23/2017 04:49 PM, Reza Arbab wrote: >> On Tue, May 23, 2017 at 03:05:08PM -0500, Michael Bringmann wrote: >>> On 05/23/2017 10:52 AM, Reza Arbab wrote: >>>> On Tue, May 23, 2017 at 10:15:44AM -0500, Michael Bringmann wrote: >>>>> +static void setup_nodes(void) >>>>> +{ >>>>> + int i, l = 32 /* MAX_NUMNODES */; >>>>> + >>>>> + for (i = 0; i < l; i++) { >>>>> + if (!node_possible(i)) { >>>>> + setup_node_data(i, 0, 0); >>>>> + node_set(i, node_possible_map); >>>>> + } >>>>> + } >>>>> +} >>>> >>>> This seems to be a workaround for 3af229f2071f ("powerpc/numa: Reset node_possible_map to only node_online_map"). >>> >>> They may be related, but that commit is not a replacement. The above patch ensures that >>> there are enough of the nodes initialized at startup to allow for memory hot-add into a >>> node that was not used at boot. (See 'setup_node_data' function in 'numa.c'.) That and >>> recording that the node was initialized. >> >> Is it really necessary to preinitialize these empty nodes using setup_node_data()? When you do memory hotadd into a node that was not used at boot, the node data already gets set up by >> >> add_memory >> add_memory_resource >> hotadd_new_pgdat >> arch_alloc_nodedata <-- allocs the pg_data_t >> ... >> free_area_init_node <-- sets NODE_DATA(nid)->node_id, etc. >> >> Removing setup_node_data() from that loop leaves only the call to node_set(). If 3af229f2071f (which reduces node_possible_map) was reverted, you wouldn't need to do that either. > >With or without 3af229f2071f, we would still need to add something, somewhere to add new >bits to the 'node_possible_map'. That is not being done. Without 3af229f2071f, those bits would already BE set in node_possible_map. You wouldn't have to do anything.
I will get a log based on the latest 4.12 kernel to show what happens one way or the other, with this patch removed. On 05/24/2017 09:36 AM, Reza Arbab wrote: > On Tue, May 23, 2017 at 05:44:23PM -0500, Michael Bringmann wrote: >> On 05/23/2017 04:49 PM, Reza Arbab wrote: >>> On Tue, May 23, 2017 at 03:05:08PM -0500, Michael Bringmann wrote: >>>> On 05/23/2017 10:52 AM, Reza Arbab wrote: >>>>> On Tue, May 23, 2017 at 10:15:44AM -0500, Michael Bringmann wrote: >>>>>> +static void setup_nodes(void) >>>>>> +{ >>>>>> + int i, l = 32 /* MAX_NUMNODES */; >>>>>> + >>>>>> + for (i = 0; i < l; i++) { >>>>>> + if (!node_possible(i)) { >>>>>> + setup_node_data(i, 0, 0); >>>>>> + node_set(i, node_possible_map); >>>>>> + } >>>>>> + } >>>>>> +} >>>>> >>>>> This seems to be a workaround for 3af229f2071f ("powerpc/numa: Reset node_possible_map to only node_online_map"). >>>> >>>> They may be related, but that commit is not a replacement. The above patch ensures that >>>> there are enough of the nodes initialized at startup to allow for memory hot-add into a >>>> node that was not used at boot. (See 'setup_node_data' function in 'numa.c'.) That and >>>> recording that the node was initialized. >>> >>> Is it really necessary to preinitialize these empty nodes using setup_node_data()? When you do memory hotadd into a node that was not used at boot, the node data already gets set up by >>> >>> add_memory >>> add_memory_resource >>> hotadd_new_pgdat >>> arch_alloc_nodedata <-- allocs the pg_data_t >>> ... >>> free_area_init_node <-- sets NODE_DATA(nid)->node_id, etc. >>> >>> Removing setup_node_data() from that loop leaves only the call to node_set(). If 3af229f2071f (which reduces node_possible_map) was reverted, you wouldn't need to do that either. >> >> With or without 3af229f2071f, we would still need to add something, somewhere to add new >> bits to the 'node_possible_map'. That is not being done. > > Without 3af229f2071f, those bits would already BE set in node_possible_map. You wouldn't have to do anything. >
On 05/24/2017 06:19 AM, Michael Ellerman wrote: > Michael Bringmann <mwb@linux.vnet.ibm.com> writes: > >> On 05/23/2017 04:49 PM, Reza Arbab wrote: >>> On Tue, May 23, 2017 at 03:05:08PM -0500, Michael Bringmann wrote: >>>> On 05/23/2017 10:52 AM, Reza Arbab wrote: >>>>> On Tue, May 23, 2017 at 10:15:44AM -0500, Michael Bringmann wrote: >>>>>> +static void setup_nodes(void) >>>>>> +{ >>>>>> + int i, l = 32 /* MAX_NUMNODES */; >>>>>> + >>>>>> + for (i = 0; i < l; i++) { >>>>>> + if (!node_possible(i)) { >>>>>> + setup_node_data(i, 0, 0); >>>>>> + node_set(i, node_possible_map); >>>>>> + } >>>>>> + } >>>>>> +} >>>>> >>>>> This seems to be a workaround for 3af229f2071f ("powerpc/numa: Reset node_possible_map to only node_online_map"). >>>> >>>> They may be related, but that commit is not a replacement. The above patch ensures that >>>> there are enough of the nodes initialized at startup to allow for memory hot-add into a >>>> node that was not used at boot. (See 'setup_node_data' function in 'numa.c'.) That and >>>> recording that the node was initialized. >>> >>> Is it really necessary to preinitialize these empty nodes using setup_node_data()? When you do memory hotadd into a node that was not used at boot, the node data already gets set up by >>> >>> add_memory >>> add_memory_resource >>> hotadd_new_pgdat >>> arch_alloc_nodedata <-- allocs the pg_data_t >>> ... >>> free_area_init_node <-- sets NODE_DATA(nid)->node_id, etc. >>> >>> Removing setup_node_data() from that loop leaves only the call to node_set(). If 3af229f2071f (which reduces node_possible_map) was reverted, you wouldn't need to do that either. >> >> With or without 3af229f2071f, we would still need to add something, somewhere to add new >> bits to the 'node_possible_map'. That is not being done. > > You mustn't add bits to the possible map after boot. > > That's its purpose, to tell you what nodes could ever *possibly* exist. The problem that I have been encountering is that the 'possible map' did *not* show all of the possible nodes. Rather, it showed only the nodes that were assigned memory at boot-up. If more memory were hot-added to the kernel, it could be assigned into one of the nodes that were skipped at boot. However, nothing was updating the 'node_possible_map' correctly in the kernel memory code. Reza pointed out a code change in commit 3af229f2071f that has not made it into the 4.12 checkout i.e. removing the instruction that reduces the node_possible_map. This may well be a suitable replacement for the code that I have here, and I will test it here next. > > cheers > > Later.
Michael Bringmann <mwb@linux.vnet.ibm.com> writes: > On 05/24/2017 06:19 AM, Michael Ellerman wrote: >> Michael Bringmann <mwb@linux.vnet.ibm.com> writes: >>> >>> With or without 3af229f2071f, we would still need to add something, somewhere to add new >>> bits to the 'node_possible_map'. That is not being done. >> >> You mustn't add bits to the possible map after boot. >> >> That's its purpose, to tell you what nodes could ever *possibly* exist. > > The problem that I have been encountering is that the 'possible map' did *not* > show all of the possible nodes. OK so how did that happen? The commit message for 3af229f2071f says: In practice, we never see a system with 256 NUMA nodes, and in fact, we do not support node hotplug on power in the first place, so the nodes ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ that are online when we come up are the nodes that will be present for the lifetime of this kernel. Is that no longer true? cheers
On 05/25/2017 01:19 AM, Michael Ellerman wrote: > Michael Bringmann <mwb@linux.vnet.ibm.com> writes: > >> On 05/24/2017 06:19 AM, Michael Ellerman wrote: >>> Michael Bringmann <mwb@linux.vnet.ibm.com> writes: >>>> >>>> With or without 3af229f2071f, we would still need to add something, somewhere to add new >>>> bits to the 'node_possible_map'. That is not being done. >>> >>> You mustn't add bits to the possible map after boot. >>> >>> That's its purpose, to tell you what nodes could ever *possibly* exist. >> >> The problem that I have been encountering is that the 'possible map' did *not* >> show all of the possible nodes. > > OK so how did that happen? > > The commit message for 3af229f2071f says: > > In practice, we never see a system with 256 NUMA nodes, and in fact, we > do not support node hotplug on power in the first place, so the nodes > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > that are online when we come up are the nodes that will be present for > the lifetime of this kernel. > > Is that no longer true? Take a look at the last part of commit 3af229f2071f for file numa.c. It undoes a piece of code that restricts the 'node possible map', created earlier, to the set of online nodes. That piece of code has not made it into the mainline, at least not into 4.12. I am testing to verify whether it is sufficient for my configuration now. > > cheers > Regards.
On Thu, May 25, 2017 at 04:19:53PM +1000, Michael Ellerman wrote: >The commit message for 3af229f2071f says: > > In practice, we never see a system with 256 NUMA nodes, and in fact, we > do not support node hotplug on power in the first place, so the nodes > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > that are online when we come up are the nodes that will be present for > the lifetime of this kernel. > >Is that no longer true? I don't know what the reasoning behind that statement was at the time, but as far as I can tell, the only thing missing for node hotplug now is Balbir's patchset [1]. He fixes the resource issue which motivated 3af229f2071f and reverts it. With that set, I can instantiate a new numa node just by doing add_memory(nid, ...) where nid doesn't currently exist. [1] https://lkml.kernel.org/r/1479253501-26261-1-git-send-email-bsingharora@gmail.com
On 05/25/2017 10:10 AM, Reza Arbab wrote: > On Thu, May 25, 2017 at 04:19:53PM +1000, Michael Ellerman wrote: >> The commit message for 3af229f2071f says: >> >> In practice, we never see a system with 256 NUMA nodes, and in fact, we >> do not support node hotplug on power in the first place, so the nodes >> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >> that are online when we come up are the nodes that will be present for >> the lifetime of this kernel. >> >> Is that no longer true? > > I don't know what the reasoning behind that statement was at the time, but as far as I can tell, the only thing missing for node hotplug now is Balbir's patchset [1]. He fixes the resource issue which motivated 3af229f2071f and reverts it. > > With that set, I can instantiate a new numa node just by doing add_memory(nid, ...) where nid doesn't currently exist. > > [1] https://lkml.kernel.org/r/1479253501-26261-1-git-send-email-bsingharora@gmail.com > Yes, the change to 'numa.c' looks to be sufficient for my needs as well.
On Thu, 25 May 2017 10:10:11 -0500 Reza Arbab <arbab@linux.vnet.ibm.com> wrote: > On Thu, May 25, 2017 at 04:19:53PM +1000, Michael Ellerman wrote: > >The commit message for 3af229f2071f says: > > > > In practice, we never see a system with 256 NUMA nodes, and in fact, we > > do not support node hotplug on power in the first place, so the nodes > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > that are online when we come up are the nodes that will be present for > > the lifetime of this kernel. > > > >Is that no longer true? > > I don't know what the reasoning behind that statement was at the time, > but as far as I can tell, the only thing missing for node hotplug now is > Balbir's patchset [1]. He fixes the resource issue which motivated > 3af229f2071f and reverts it. > > With that set, I can instantiate a new numa node just by doing > add_memory(nid, ...) where nid doesn't currently exist. > > [1] https://lkml.kernel.org/r/1479253501-26261-1-git-send-email-bsingharora@gmail.com > I guess I should try and revive that patchset. One of the suggestions of then was to limit maximum possible nodes in firmware, but I'm double checking to see if we can do that in a well defined manner. Balbir Singh
Reza Arbab <arbab@linux.vnet.ibm.com> writes: > On Thu, May 25, 2017 at 04:19:53PM +1000, Michael Ellerman wrote: >>The commit message for 3af229f2071f says: >> >> In practice, we never see a system with 256 NUMA nodes, and in fact, we >> do not support node hotplug on power in the first place, so the nodes >> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >> that are online when we come up are the nodes that will be present for >> the lifetime of this kernel. >> >>Is that no longer true? > > I don't know what the reasoning behind that statement was at the time, > but as far as I can tell, the only thing missing for node hotplug now is > Balbir's patchset [1]. He fixes the resource issue which motivated > 3af229f2071f and reverts it. > > With that set, I can instantiate a new numa node just by doing > add_memory(nid, ...) where nid doesn't currently exist. But does that actually happen on any real system? cheers
I am running into this problem on PowerPC systems where Balbir's patch set was targeted. So, yes, I do need to be able to add/enable a new numa node during system execution in cases where more resources (memory, virtual processors) are added to the system dynamically. On 05/25/2017 10:46 PM, Michael Ellerman wrote: > Reza Arbab <arbab@linux.vnet.ibm.com> writes: > >> On Thu, May 25, 2017 at 04:19:53PM +1000, Michael Ellerman wrote: >>> The commit message for 3af229f2071f says: >>> >>> In practice, we never see a system with 256 NUMA nodes, and in fact, we >>> do not support node hotplug on power in the first place, so the nodes >>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>> that are online when we come up are the nodes that will be present for >>> the lifetime of this kernel. >>> >>> Is that no longer true? >> >> I don't know what the reasoning behind that statement was at the time, >> but as far as I can tell, the only thing missing for node hotplug now is >> Balbir's patchset [1]. He fixes the resource issue which motivated >> 3af229f2071f and reverts it. >> >> With that set, I can instantiate a new numa node just by doing >> add_memory(nid, ...) where nid doesn't currently exist. > > But does that actually happen on any real system? > > cheers > >
On Fri, May 26, 2017 at 01:46:58PM +1000, Michael Ellerman wrote: >Reza Arbab <arbab@linux.vnet.ibm.com> writes: > >> On Thu, May 25, 2017 at 04:19:53PM +1000, Michael Ellerman wrote: >>>The commit message for 3af229f2071f says: >>> >>> In practice, we never see a system with 256 NUMA nodes, and in fact, we >>> do not support node hotplug on power in the first place, so the nodes >>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>> that are online when we come up are the nodes that will be present for >>> the lifetime of this kernel. >>> >>>Is that no longer true? >> >> I don't know what the reasoning behind that statement was at the time, >> but as far as I can tell, the only thing missing for node hotplug now is >> Balbir's patchset [1]. He fixes the resource issue which motivated >> 3af229f2071f and reverts it. >> >> With that set, I can instantiate a new numa node just by doing >> add_memory(nid, ...) where nid doesn't currently exist. > >But does that actually happen on any real system? I don't know if anything currently tries to do this. My interest in having this working is so that in the future, our coherent gpu memory could be added as a distinct node by the device driver.
Reza Arbab <arbab@linux.vnet.ibm.com> writes: > On Fri, May 26, 2017 at 01:46:58PM +1000, Michael Ellerman wrote: >>Reza Arbab <arbab@linux.vnet.ibm.com> writes: >> >>> On Thu, May 25, 2017 at 04:19:53PM +1000, Michael Ellerman wrote: >>>>The commit message for 3af229f2071f says: >>>> >>>> In practice, we never see a system with 256 NUMA nodes, and in fact, we >>>> do not support node hotplug on power in the first place, so the nodes >>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>> that are online when we come up are the nodes that will be present for >>>> the lifetime of this kernel. >>>> >>>>Is that no longer true? >>> >>> I don't know what the reasoning behind that statement was at the time, >>> but as far as I can tell, the only thing missing for node hotplug now is >>> Balbir's patchset [1]. He fixes the resource issue which motivated >>> 3af229f2071f and reverts it. >>> >>> With that set, I can instantiate a new numa node just by doing >>> add_memory(nid, ...) where nid doesn't currently exist. >> >>But does that actually happen on any real system? > > I don't know if anything currently tries to do this. My interest in > having this working is so that in the future, our coherent gpu memory > could be added as a distinct node by the device driver. Sure. If/when that happens, we would hopefully still have some way to limit the size of the possible map. That would ideally be a firmware property that tells us the maximum number of GPUs that might be hot-added, or we punt and cap it at some "sane" maximum number. But until that happens it's silly to say we can have up to 256 nodes when in practice most of our systems have 8 or less. So I'm still waiting for an explanation from Michael B on how he's seeing this bug in practice. cheers
On 05/29/2017 12:32 AM, Michael Ellerman wrote: > Reza Arbab <arbab@linux.vnet.ibm.com> writes: > >> On Fri, May 26, 2017 at 01:46:58PM +1000, Michael Ellerman wrote: >>> Reza Arbab <arbab@linux.vnet.ibm.com> writes: >>> >>>> On Thu, May 25, 2017 at 04:19:53PM +1000, Michael Ellerman wrote: >>>>> The commit message for 3af229f2071f says: >>>>> >>>>> In practice, we never see a system with 256 NUMA nodes, and in fact, we >>>>> do not support node hotplug on power in the first place, so the nodes >>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>>> that are online when we come up are the nodes that will be present for >>>>> the lifetime of this kernel. >>>>> >>>>> Is that no longer true? >>>> >>>> I don't know what the reasoning behind that statement was at the time, >>>> but as far as I can tell, the only thing missing for node hotplug now is >>>> Balbir's patchset [1]. He fixes the resource issue which motivated >>>> 3af229f2071f and reverts it. >>>> >>>> With that set, I can instantiate a new numa node just by doing >>>> add_memory(nid, ...) where nid doesn't currently exist. >>> >>> But does that actually happen on any real system? >> >> I don't know if anything currently tries to do this. My interest in >> having this working is so that in the future, our coherent gpu memory >> could be added as a distinct node by the device driver. > > Sure. If/when that happens, we would hopefully still have some way to > limit the size of the possible map. > > That would ideally be a firmware property that tells us the maximum > number of GPUs that might be hot-added, or we punt and cap it at some > "sane" maximum number. > > But until that happens it's silly to say we can have up to 256 nodes > when in practice most of our systems have 8 or less. > > So I'm still waiting for an explanation from Michael B on how he's > seeing this bug in practice. I already answered this in an earlier message. I will give an example. * Let there be a configuration with nodes (0, 4-5, 8) that boots with 1 VP and 10G of memory in a shared processor configuration. * At boot time, 4 nodes are put into the possible map by the PowerPC boot code. * Subsequently, the NUMA code executes and puts the 10G memory into nodes 4 & 5. No memory goes into Node 0. So we now have 2 nodes in the node_online_map. * The VP and its threads get assigned to Node 4. * Then when 'initmem_init()' in 'powerpc/numa.c' executes the instruction, node_and(node_possible_map, node_possible_map, node_online_map); the content of the node_possible_map is reduced to nodes 4-5. * Later on we hot-add 90G of memory to the system. It tries to put the memory into nodes 0, 4-5, 8 based on the memory association map. We should see memory put into all 4 nodes. However, since we have reduced the 'node_possible_map' to only nodes 4 & 5, we can now only put memory into 2 of the configured nodes. # We want to be able to put memory into all 4 nodes via hot-add operations, not only the nodes that 'survive' boot time initialization. We could make a number of changes to ensure that all of the nodes in the initial configuration provided by the pHyp can be used, but this one appears to be the simplest, only using resources requested by the pHyp at boot -- even if those resource are not used immediately. > > cheers > Regards, Michael
Michael Bringmann <mwb@linux.vnet.ibm.com> writes: > On 05/29/2017 12:32 AM, Michael Ellerman wrote: >> Reza Arbab <arbab@linux.vnet.ibm.com> writes: >> >>> On Fri, May 26, 2017 at 01:46:58PM +1000, Michael Ellerman wrote: >>>> Reza Arbab <arbab@linux.vnet.ibm.com> writes: >>>> >>>>> On Thu, May 25, 2017 at 04:19:53PM +1000, Michael Ellerman wrote: >>>>>> The commit message for 3af229f2071f says: >>>>>> >>>>>> In practice, we never see a system with 256 NUMA nodes, and in fact, we >>>>>> do not support node hotplug on power in the first place, so the nodes >>>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>>>> that are online when we come up are the nodes that will be present for >>>>>> the lifetime of this kernel. >>>>>> >>>>>> Is that no longer true? >>>>> >>>>> I don't know what the reasoning behind that statement was at the time, >>>>> but as far as I can tell, the only thing missing for node hotplug now is >>>>> Balbir's patchset [1]. He fixes the resource issue which motivated >>>>> 3af229f2071f and reverts it. >>>>> >>>>> With that set, I can instantiate a new numa node just by doing >>>>> add_memory(nid, ...) where nid doesn't currently exist. >>>> >>>> But does that actually happen on any real system? >>> >>> I don't know if anything currently tries to do this. My interest in >>> having this working is so that in the future, our coherent gpu memory >>> could be added as a distinct node by the device driver. >> >> Sure. If/when that happens, we would hopefully still have some way to >> limit the size of the possible map. >> >> That would ideally be a firmware property that tells us the maximum >> number of GPUs that might be hot-added, or we punt and cap it at some >> "sane" maximum number. >> >> But until that happens it's silly to say we can have up to 256 nodes >> when in practice most of our systems have 8 or less. >> >> So I'm still waiting for an explanation from Michael B on how he's >> seeing this bug in practice. > > I already answered this in an earlier message. Which one? I must have missed it. > I will give an example. > > * Let there be a configuration with nodes (0, 4-5, 8) that boots with 1 VP > and 10G of memory in a shared processor configuration. > * At boot time, 4 nodes are put into the possible map by the PowerPC boot > code. I'm pretty sure we never add nodes to the possible map, it starts out with MAX_NUMNODES possible and that's it. Do you actually see mention of nodes 0 and 8 in the dmesg? What does it say? > * Subsequently, the NUMA code executes and puts the 10G memory into nodes > 4 & 5. No memory goes into Node 0. So we now have 2 nodes in the > node_online_map. > * The VP and its threads get assigned to Node 4. > * Then when 'initmem_init()' in 'powerpc/numa.c' executes the instruction, > node_and(node_possible_map, node_possible_map, node_online_map); > the content of the node_possible_map is reduced to nodes 4-5. > * Later on we hot-add 90G of memory to the system. It tries to put the > memory into nodes 0, 4-5, 8 based on the memory association map. We > should see memory put into all 4 nodes. However, since we have reduced > the 'node_possible_map' to only nodes 4 & 5, we can now only put memory > into 2 of the configured nodes. Right. So it's not that you're hot adding memory into a previously unseen node as you implied in earlier mails. > # We want to be able to put memory into all 4 nodes via hot-add operations, > not only the nodes that 'survive' boot time initialization. We could > make a number of changes to ensure that all of the nodes in the initial > configuration provided by the pHyp can be used, but this one appears to > be the simplest, only using resources requested by the pHyp at boot -- > even if those resource are not used immediately. I don't think that's what the patch does. It just marks 32 (!?) nodes as online. Or if you're talking about reverting 3af229f2071f that leaves you with 256 possible nodes. Both of which are wasteful. The right fix is to make sure any nodes which are present at boot remain in the possible map, even if they don't have memory/CPUs assigned at boot. What does your device tree look like? Can you send us the output of: $ lsprop /proc/device-tree cheers
On 06/01/2017 04:36 AM, Michael Ellerman wrote: > Michael Bringmann <mwb@linux.vnet.ibm.com> writes: > >> On 05/29/2017 12:32 AM, Michael Ellerman wrote: >>> Reza Arbab <arbab@linux.vnet.ibm.com> writes: >>> >>>> On Fri, May 26, 2017 at 01:46:58PM +1000, Michael Ellerman wrote: >>>>> Reza Arbab <arbab@linux.vnet.ibm.com> writes: >>>>> >>>>>> On Thu, May 25, 2017 at 04:19:53PM +1000, Michael Ellerman wrote: >>>>>>> The commit message for 3af229f2071f says: >>>>>>> >>>>>>> In practice, we never see a system with 256 NUMA nodes, and in fact, we >>>>>>> do not support node hotplug on power in the first place, so the nodes >>>>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>>>>> that are online when we come up are the nodes that will be present for >>>>>>> the lifetime of this kernel. >>>>>>> >>>>>>> Is that no longer true? >>>>>> >>>>>> I don't know what the reasoning behind that statement was at the time, >>>>>> but as far as I can tell, the only thing missing for node hotplug now is >>>>>> Balbir's patchset [1]. He fixes the resource issue which motivated >>>>>> 3af229f2071f and reverts it. >>>>>> >>>>>> With that set, I can instantiate a new numa node just by doing >>>>>> add_memory(nid, ...) where nid doesn't currently exist. >>>>> >>>>> But does that actually happen on any real system? >>>> >>>> I don't know if anything currently tries to do this. My interest in >>>> having this working is so that in the future, our coherent gpu memory >>>> could be added as a distinct node by the device driver. >>> >>> Sure. If/when that happens, we would hopefully still have some way to >>> limit the size of the possible map. >>> >>> That would ideally be a firmware property that tells us the maximum >>> number of GPUs that might be hot-added, or we punt and cap it at some >>> "sane" maximum number. >>> >>> But until that happens it's silly to say we can have up to 256 nodes >>> when in practice most of our systems have 8 or less. >>> >>> So I'm still waiting for an explanation from Michael B on how he's >>> seeing this bug in practice. >> >> I already answered this in an earlier message. > > Which one? I must have missed it. > >> I will give an example. >> >> * Let there be a configuration with nodes (0, 4-5, 8) that boots with 1 VP >> and 10G of memory in a shared processor configuration. >> * At boot time, 4 nodes are put into the possible map by the PowerPC boot >> code. > > I'm pretty sure we never add nodes to the possible map, it starts out > with MAX_NUMNODES possible and that's it. Let me reword that. It enables the nodes in the possible map. > > Do you actually see mention of nodes 0 and 8 in the dmesg? When the 'numa.c' code is built with debug messages, and the system was given that configuration by pHyp, yes, I did. > > What does it say? The debug message for each core thread would be something like, removing cpu 64 from node 0 adding cpu 64 to node 8 repeated for all 8 threads of the CPU, and usually with the messages for all of the CPUs coming out intermixed on the console/dmesg log. > >> * Subsequently, the NUMA code executes and puts the 10G memory into nodes >> 4 & 5. No memory goes into Node 0. So we now have 2 nodes in the >> node_online_map. >> * The VP and its threads get assigned to Node 4. >> * Then when 'initmem_init()' in 'powerpc/numa.c' executes the instruction, >> node_and(node_possible_map, node_possible_map, node_online_map); >> the content of the node_possible_map is reduced to nodes 4-5. >> * Later on we hot-add 90G of memory to the system. It tries to put the >> memory into nodes 0, 4-5, 8 based on the memory association map. We >> should see memory put into all 4 nodes. However, since we have reduced >> the 'node_possible_map' to only nodes 4 & 5, we can now only put memory >> into 2 of the configured nodes. > > Right. So it's not that you're hot adding memory into a previously > unseen node as you implied in earlier mails. In the sense that the nodes were defined in the device tree, that is correct. In the sense that those nodes are currently deleted from node_possible_map in 'numa.c' by the instruction 'node_and(node_possible_map,node_possible_map, node_online_map);', the nodes are no longer available to place memory or CPU. >> # We want to be able to put memory into all 4 nodes via hot-add operations, >> not only the nodes that 'survive' boot time initialization. We could >> make a number of changes to ensure that all of the nodes in the initial >> configuration provided by the pHyp can be used, but this one appears to >> be the simplest, only using resources requested by the pHyp at boot -- >> even if those resource are not used immediately. > > I don't think that's what the patch does. It just marks 32 (!?) nodes as > online. Or if you're talking about reverting 3af229f2071f that leaves > you with 256 possible nodes. Both of which are wasteful> > The right fix is to make sure any nodes which are present at boot remain > in the possible map, even if they don't have memory/CPUs assigned at > boot. Okay, I can try to insert code that extracts all of the nodes from the ibm,associativity-lookup-arrays property and merge them with the nodes put into the online map from the CPUs that were found previously during boot of the powerpc code. > What does your device tree look like? Can you send us the output of: > > $ lsprop /proc/device-tree See attachment 'device-tree.log'. Note though that this boot of my test system only has 2 nodes, 0 and 2. > > cheers >
Michael Bringmann <mwb@linux.vnet.ibm.com> writes: > On 06/01/2017 04:36 AM, Michael Ellerman wrote: >> Do you actually see mention of nodes 0 and 8 in the dmesg? > > When the 'numa.c' code is built with debug messages, and the system was > given that configuration by pHyp, yes, I did. > >> What does it say? > > The debug message for each core thread would be something like, > > removing cpu 64 from node 0 > adding cpu 64 to node 8 > > repeated for all 8 threads of the CPU, and usually with the messages > for all of the CPUs coming out intermixed on the console/dmesg log. OK. I meant what do you see at boot. I'm curious how we're discovering node 0 and 8 at all if neither has any memory or CPUs assigned at boot. >> Right. So it's not that you're hot adding memory into a previously >> unseen node as you implied in earlier mails. > > In the sense that the nodes were defined in the device tree, that is correct. Where are they defined in the device tree? That's what I'm trying to understand. > In the sense that those nodes are currently deleted from node_possible_map in > 'numa.c' by the instruction 'node_and(node_possible_map,node_possible_map, > node_online_map);', the nodes are no longer available to place memory or CPU. Yeah I understand that part. > Okay, I can try to insert code that extracts all of the nodes from the > ibm,associativity-lookup-arrays property and merge them with the nodes > put into the online map from the CPUs that were found previously during > boot of the powerpc code. Hmm, will that work? Looking at PAPR it's not clear to me that it will work for nodes that have no memory assigned at boot. This property is used to duplicate the function of the “ibm,associativity” property in a /memory node. Each “assigned” LMB represented has an index valued between 0 and M-1 which is used as in index into this table to select which associativity list to use for the LMB. “unassigned” LMBs are place holders for potential DLPAR additions, for which the associativity list index is meaningless and ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ is given the reserved value of -1. This static property, need only contain values relevant for the LMBs presented in the “ibm,dynamicreconfiguration-memory” node; for a dynamic LPAR addition of a new LMB, the device tree fragment reported by the ibm,configure-connector RTAS function is a /memory node, with the inclusion of the “ibm,associativity” device tree property defined in Section C.6.2.2‚ “Properties of the Children of Root‚” on page 1059. >> What does your device tree look like? Can you send us the output of: >> >> $ lsprop /proc/device-tree Thanks. I forgot that lsprop will truncate long properties, I actually wanted to see all of the ibm,dynamic-memory property. But looking at the code I see the only place we set a nid online is if there is a CPU assigned to it: static int __init parse_numa_properties(void) { ... for_each_present_cpu(i) { ... cpu = of_get_cpu_node(i, NULL); nid = of_node_to_nid_single(cpu); ... node_set_online(nid); } Or for memory nodes (same function): for_each_node_by_type(memory, "memory") { ... nid = of_node_to_nid_single(memory); ... node_set_online(nid); ... } Or for entries in ibm,dynamic-memory that are assigned: static void __init parse_drconf_memory(struct device_node *memory) { ... for (; n != 0; --n) { ... /* skip this block if the reserved bit is set in flags (0x80) or if the block is not assigned to this partition (0x8) */ if ((drmem.flags & DRCONF_MEM_RESERVED) || !(drmem.flags & DRCONF_MEM_ASSIGNED)) continue; ... do { ... nid = of_drconf_to_nid_single(&drmem, &aa); node_set_online(nid); ... } while (--ranges); } } So I don't see from that how we can even be aware that node 0 and 8 exist at boot based on that. Maybe there's another path I'm missing though. cheers
On 06/06/2017 04:48 AM, Michael Ellerman wrote: > Michael Bringmann <mwb@linux.vnet.ibm.com> writes: >> On 06/01/2017 04:36 AM, Michael Ellerman wrote: >>> Do you actually see mention of nodes 0 and 8 in the dmesg? >> >> When the 'numa.c' code is built with debug messages, and the system was >> given that configuration by pHyp, yes, I did. >> >>> What does it say? >> >> The debug message for each core thread would be something like, >> >> removing cpu 64 from node 0 >> adding cpu 64 to node 8 >> >> repeated for all 8 threads of the CPU, and usually with the messages >> for all of the CPUs coming out intermixed on the console/dmesg log. > > OK. I meant what do you see at boot. Here is an example with nodes 0,2,6,7, node 0 starts out empty: [ 0.000000] Initmem setup node 0 [ 0.000000] NODE_DATA [mem 0x3bff7d6300-0x3bff7dffff] [ 0.000000] NODE_DATA(0) on node 7 [ 0.000000] Initmem setup node 2 [mem 0x00000000-0x13ffffffff] [ 0.000000] NODE_DATA [mem 0x13ffff6300-0x13ffffffff] [ 0.000000] Initmem setup node 6 [mem 0x1400000000-0x34afffffff] [ 0.000000] NODE_DATA [mem 0x34afff6300-0x34afffffff] [ 0.000000] Initmem setup node 7 [mem 0x34b0000000-0x3bffffffff] [ 0.000000] NODE_DATA [mem 0x3bff7cc600-0x3bff7d62ff] [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000000000-0x0000003bffffffff] [ 0.000000] DMA32 empty [ 0.000000] Normal empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 2: [mem 0x0000000000000000-0x00000013ffffffff] [ 0.000000] node 6: [mem 0x0000001400000000-0x00000034afffffff] [ 0.000000] node 7: [mem 0x00000034b0000000-0x0000003bffffffff] [ 0.000000] Could not find start_pfn for node 0 [ 0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x0000000000000000] [ 0.000000] Initmem setup node 2 [mem 0x0000000000000000-0x00000013ffffffff] [ 0.000000] Initmem setup node 6 [mem 0x0000001400000000-0x00000034afffffff] [ 0.000000] Initmem setup node 7 [mem 0x00000034b0000000-0x0000003bffffffff] [ 0.000000] percpu: Embedded 3 pages/cpu @c000003bf8000000 s155672 r0 d40936 u262144 [ 0.000000] Built 4 zonelists in Node order, mobility grouping on. Total pages: 3928320 and, [root@ltcalpine2-lp20 ~]# numactl --hardware available: 4 nodes (0,2,6-7) node 0 cpus: node 0 size: 0 MB node 0 free: 0 MB node 2 cpus: 16 17 18 19 20 21 22 23 32 33 34 35 36 37 38 39 56 57 58 59 60 61 62 63 node 2 size: 81792 MB node 2 free: 81033 MB node 6 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 24 25 26 27 28 29 30 31 40 41 42 43 44 45 46 47 node 6 size: 133743 MB node 6 free: 133097 MB node 7 cpus: 48 49 50 51 52 53 54 55 node 7 size: 29877 MB node 7 free: 29599 MB node distances: node 0 2 6 7 0: 10 40 40 40 2: 40 10 40 40 6: 40 40 10 20 7: 40 40 20 10 [root@ltcalpine2-lp20 ~]# > > I'm curious how we're discovering node 0 and 8 at all if neither has any > memory or CPUs assigned at boot. Both are in the memory associativity lookup arrays. And we are circling back to the > >>> Right. So it's not that you're hot adding memory into a previously >>> unseen node as you implied in earlier mails. >> >> In the sense that the nodes were defined in the device tree, that is correct. > > Where are they defined in the device tree? That's what I'm trying to understand. The nodes for memory are defined one time in "ibm,associativity-lookup-arrays". I wish that there was an official set of node properties in the device-tree, instead of having them be the values of other properties. > >> In the sense that those nodes are currently deleted from node_possible_map in >> 'numa.c' by the instruction 'node_and(node_possible_map,node_possible_map, >> node_online_map);', the nodes are no longer available to place memory or CPU. > > Yeah I understand that part. > >> Okay, I can try to insert code that extracts all of the nodes from the >> ibm,associativity-lookup-arrays property and merge them with the nodes >> put into the online map from the CPUs that were found previously during >> boot of the powerpc code. > > Hmm, will that work? The nodes are defined in the associativity lookup array, so they have at least been reserved for us by the pHyp. On the other hand, if we are only to use nodes that have resources at boot, why are there extra node values specified? What I am not 100% clear on -- and why I preferred letting all possible nodes originally defined, still be possible for subsequent hot-add operations -- is whether the nodes to be used for hot-added CPUs would always be a subset of the nodes used for hot-added memory. * The hot-added CPUs in Shared CPU configurations may be mapped to nodes by the value returned to the kernel by the VPHN hcall. * So far in my tests, this has not been a problem, but I could not be positive from the PAPR. > > Looking at PAPR it's not clear to me that it will work for nodes that > have no memory assigned at boot. > > This property is used to duplicate the function of the > “ibm,associativity” property in a /memory node. Each “assigned” LMB > represented has an index valued between 0 and M-1 which is used as in > index into this table to select which associativity list to use for > the LMB. “unassigned” LMBs are place holders for potential DLPAR > additions, for which the associativity list index is meaningless and > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > is given the reserved value of -1. This static property, need only > contain values relevant for the LMBs presented in the > “ibm,dynamicreconfiguration-memory” node; for a dynamic LPAR addition > of a new LMB, the device tree fragment reported by the > ibm,configure-connector RTAS function is a /memory node, with the > inclusion of the “ibm,associativity” device tree property defined in > Section C.6.2.2‚ “Properties of the Children of Root‚” on page 1059. I don't see any place that builds new /memory nodes in conjunction with hot-added memory. The code for powerpc treats the definitions provided by 'ibm,dynamic-reconfiguration-memory' as the primary reference wherever hot-added memory comes into play. It looks like the '/memory' properties are backups or used for the 'pseries' according to one comment. > >>> What does your device tree look like? Can you send us the output of: >>> >>> $ lsprop /proc/device-tree > > Thanks. I forgot that lsprop will truncate long properties, I actually > wanted to see all of the ibm,dynamic-memory property. > > But looking at the code I see the only place we set a nid online is if > there is a CPU assigned to it: > > static int __init parse_numa_properties(void) > { > ... > for_each_present_cpu(i) { > ... > cpu = of_get_cpu_node(i, NULL); > nid = of_node_to_nid_single(cpu); > ... > node_set_online(nid); > } > > Or for memory nodes (same function): > > for_each_node_by_type(memory, "memory") { > ... > nid = of_node_to_nid_single(memory); > ... > node_set_online(nid); > ... > } > > Or for entries in ibm,dynamic-memory that are assigned: > > static void __init parse_drconf_memory(struct device_node *memory) > { > ... > for (; n != 0; --n) { > ... > /* skip this block if the reserved bit is set in flags (0x80) > or if the block is not assigned to this partition (0x8) */ > if ((drmem.flags & DRCONF_MEM_RESERVED) > || !(drmem.flags & DRCONF_MEM_ASSIGNED)) > continue; > > ... > do { > ... > nid = of_drconf_to_nid_single(&drmem, &aa); > node_set_online(nid); > ... > } while (--ranges); > } > } > > > So I don't see from that how we can even be aware that node 0 and 8 > exist at boot based on that. Maybe there's another path I'm missing > though. We don't 'fill in' the nodes, but we are aware that they exist per the 'ibm,associativity-lookup-arrays' or the responsed provided by the pHyp to the VPHN hcall. We don't associate either of these resources to them, but does that mean that the nodes do not exist? The code currently says that only nodes booted with resources "exist" i.e. it can't hot-add new nodes, but is that a just a problem of the kernel implementation? I think so. However, this is the problem for users running systems that hot-add a lot of resources are concerned. They see the associativity arrays (and 'hypinfo' table internal to the pHyp), and they ask why the kernel only records new resources into the boot-time nodes, while pHyp appears to distribute across all of the memory nodes specified to the LPAR of the kernel at boot. I think that all of those nodes specified by the pHyp should exist to the kernel, and that we are trying to find the best way to make them visible here. > > cheers > >
Michael Bringmann <mwb@linux.vnet.ibm.com> writes: > On 06/06/2017 04:48 AM, Michael Ellerman wrote: >> Michael Bringmann <mwb@linux.vnet.ibm.com> writes: >>> On 06/01/2017 04:36 AM, Michael Ellerman wrote: >>>> Do you actually see mention of nodes 0 and 8 in the dmesg? >>> >>> When the 'numa.c' code is built with debug messages, and the system was >>> given that configuration by pHyp, yes, I did. >>> >>>> What does it say? >>> >>> The debug message for each core thread would be something like, >>> >>> removing cpu 64 from node 0 >>> adding cpu 64 to node 8 >>> >>> repeated for all 8 threads of the CPU, and usually with the messages >>> for all of the CPUs coming out intermixed on the console/dmesg log. >> >> OK. I meant what do you see at boot. > > Here is an example with nodes 0,2,6,7, node 0 starts out empty: > > [ 0.000000] Initmem setup node 0 > [ 0.000000] NODE_DATA [mem 0x3bff7d6300-0x3bff7dffff] > [ 0.000000] NODE_DATA(0) on node 7 > [ 0.000000] Initmem setup node 2 [mem 0x00000000-0x13ffffffff] > [ 0.000000] NODE_DATA [mem 0x13ffff6300-0x13ffffffff] > [ 0.000000] Initmem setup node 6 [mem 0x1400000000-0x34afffffff] > [ 0.000000] NODE_DATA [mem 0x34afff6300-0x34afffffff] > [ 0.000000] Initmem setup node 7 [mem 0x34b0000000-0x3bffffffff] > [ 0.000000] NODE_DATA [mem 0x3bff7cc600-0x3bff7d62ff] > > [ 0.000000] Zone ranges: > [ 0.000000] DMA [mem 0x0000000000000000-0x0000003bffffffff] > [ 0.000000] DMA32 empty > [ 0.000000] Normal empty > [ 0.000000] Movable zone start for each node > [ 0.000000] Early memory node ranges > [ 0.000000] node 2: [mem 0x0000000000000000-0x00000013ffffffff] > [ 0.000000] node 6: [mem 0x0000001400000000-0x00000034afffffff] > [ 0.000000] node 7: [mem 0x00000034b0000000-0x0000003bffffffff] > [ 0.000000] Could not find start_pfn for node 0 > [ 0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x0000000000000000] > [ 0.000000] Initmem setup node 2 [mem 0x0000000000000000-0x00000013ffffffff] > [ 0.000000] Initmem setup node 6 [mem 0x0000001400000000-0x00000034afffffff] > [ 0.000000] Initmem setup node 7 [mem 0x00000034b0000000-0x0000003bffffffff] > [ 0.000000] percpu: Embedded 3 pages/cpu @c000003bf8000000 s155672 r0 d40936 u262144 > [ 0.000000] Built 4 zonelists in Node order, mobility grouping on. Total pages: 3928320 > > and, > > [root@ltcalpine2-lp20 ~]# numactl --hardware > available: 4 nodes (0,2,6-7) > node 0 cpus: > node 0 size: 0 MB > node 0 free: 0 MB > node 2 cpus: 16 17 18 19 20 21 22 23 32 33 34 35 36 37 38 39 56 57 58 59 60 61 62 63 > node 2 size: 81792 MB > node 2 free: 81033 MB > node 6 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 24 25 26 27 28 29 30 31 40 41 42 43 44 45 46 47 > node 6 size: 133743 MB > node 6 free: 133097 MB > node 7 cpus: 48 49 50 51 52 53 54 55 > node 7 size: 29877 MB > node 7 free: 29599 MB > node distances: > node 0 2 6 7 > 0: 10 40 40 40 > 2: 40 10 40 40 > 6: 40 40 10 20 > 7: 40 40 20 10 > [root@ltcalpine2-lp20 ~]# What kernel is that running? And can you show me the full ibm,dynamic-memory and lookup-arrays properties for that system? cheers
Red Hat Enterprise Linux Server 7.3 (Maipo) Kernel 4.12.0-rc3.wi91275_054c_060106.ppc64le+ on an ppc64le ltcalpine2-lp20 login: root Password: Last login: Wed Jun 7 11:03:12 from oc1554177480.austin.ibm.com [root@ltcalpine2-lp20 ~]# numactl -H available: 3 nodes (0,2-3) node 0 cpus: node 0 size: 0 MB node 0 free: 0 MB node 2 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 48 49 50 51 52 53 54 55 node 2 size: 188668 MB node 2 free: 187903 MB node 3 cpus: 40 41 42 43 44 45 46 47 56 57 58 59 60 61 62 63 node 3 size: 56261 MB node 3 free: 55324 MB node distances: node 0 2 3 0: 10 40 40 2: 40 10 20 3: 40 20 10 [root@ltcalpine2-lp20 ~]# cd /proc/device-tree/ibm,dynamic-reconfiguration-memory [root@ltcalpine2-lp20 ibm,dynamic-reconfiguration-memory]# lsprop ibm,dynamic-memory ibm,dynamic-memory 0000059e 00000000 20000000 80000002 00000000 00000001 00000008 00000000 30000000 80000003 00000000 00000001 00000008 00000000 40000000 80000004 00000000 00000001 00000008 00000000 50000000 80000005 00000000 00000001 00000008 00000000 60000000 80000006 00000000 00000001 00000008 00000000 [34516 bytes total] [root@ltcalpine2-lp20 ibm,dynamic-reconfiguration-memory]# lsprop ibm,associativity-lookup-arrays ibm,associativity-lookup-arrays 00000003 00000004 00000000 00000000 00000000 00000000 00000000 00000001 00000002 00000002 00000000 00000001 00000003 00000003 [root@ltcalpine2-lp20 ibm,dynamic-reconfiguration-memory]# cat ~/dmesg.log [ 0.000000] Linux version 4.12.0-rc3.wi91275_054c_060106.ppc64le+ (root@ltcalpine2-lp20.aus.stglabs.ibm.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #28 SMP Tue Jun 6 16:05:01 EDT 2017 [ 0.000000] Found initrd at 0xc00000000ce00000:0xc00000000eeb4b12 [ 0.000000] Using pSeries machine description [ 0.000000] bootconsole [udbg0] enabled [ 0.000000] Partition configured for 256 cpus. [ 0.000000] CPU maps initialized for 8 threads per core [ 0.000000] (thread shift is 3) [ 0.000000] Freed 4784128 bytes for unused pacas [ 0.000000] ----------------------------------------------------- [ 0.000000] ppc64_pft_size = 0x20 [ 0.000000] phys_mem_size = 0x3c00000000 [ 0.000000] dcache_bsize = 0x80 [ 0.000000] icache_bsize = 0x80 [ 0.000000] cpu_features = 0x27fc7aec18500249 [ 0.000000] possible = 0x7fffffff18500649 [ 0.000000] always = 0x0000000018100040 [ 0.000000] cpu_user_features = 0xdc0065c2 0xef000000 [ 0.000000] mmu_features = 0x7c006001 [ 0.000000] firmware_features = 0x00000003c45bfc57 [ 0.000000] htab_hash_mask = 0x1ffffff [ 0.000000] ----------------------------------------------------- [ 0.000000] numa: NODE_DATA [mem 0x3bff7d5f00-0x3bff7dffff] [ 0.000000] numa: NODE_DATA(0) on node 3 [ 0.000000] numa: NODE_DATA [mem 0x2e1fff5f00-0x2e1fffffff] [ 0.000000] numa: NODE_DATA [mem 0x3bff7cbe00-0x3bff7d5eff] [ 0.000000] Section 15357 and 15359 (node 3) have a circular dependency on usemap and pgdat allocations [ 0.000000] PPC64 nvram contains 15360 bytes [ 0.000000] Top of RAM: 0x3c00000000, Total RAM: 0x3c00000000 [ 0.000000] Memory hole size: 0MB [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000000000-0x0000003bffffffff] [ 0.000000] DMA32 empty [ 0.000000] Normal empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 2: [mem 0x0000000000000000-0x0000002e1fffffff] [ 0.000000] node 3: [mem 0x0000002e20000000-0x0000003bffffffff] [ 0.000000] Could not find start_pfn for node 0 [ 0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x0000000000000000] [ 0.000000] On node 0 totalpages: 0 [ 0.000000] Initmem setup node 2 [mem 0x0000000000000000-0x0000002e1fffffff] [ 0.000000] On node 2 totalpages: 3022848 [ 0.000000] DMA zone: 2952 pages used for memmap [ 0.000000] DMA zone: 0 pages reserved [ 0.000000] DMA zone: 3022848 pages, LIFO batch:1 [ 0.000000] Initmem setup node 3 [mem 0x0000002e20000000-0x0000003bffffffff] [ 0.000000] On node 3 totalpages: 909312 [ 0.000000] DMA zone: 888 pages used for memmap [ 0.000000] DMA zone: 0 pages reserved [ 0.000000] DMA zone: 909312 pages, LIFO batch:1 [root@ltcalpine2-lp20 ibm,dynamic-reconfiguration-memory]# Welcome to SUSE Linux Enterprise Server 12 SP3 Beta2 (ppc64le) - Kernel 4.4.63-2.1.wi91275.sp3.055a_27-default (hvc0). ltcalpine2-lp9 login: root Password: Last login: Mon Jun 5 11:43:45 from 9.53.92.157 ltcalpine2-lp9:ß # numactl -H available: 4 nodes (0-1,6-7) node 0 cpus: node 0 size: 20512 MB node 0 free: 20371 MB node 1 cpus: node 1 size: 2815 MB node 1 free: 2706 MB node 6 cpus: 0 1 2 3 4 5 6 7 node 6 size: 134783 MB node 6 free: 134068 MB node 7 cpus: node 7 size: 87210 MB node 7 free: 87102 MB node distances: node 0 1 6 7 0: 10 20 40 40 1: 20 10 40 40 6: 40 40 10 20 7: 40 40 20 10 ltcalpine2-lp9:~ # cd ltcalpine2-lp9:~ # cd /proc/device-tree/ibm,dynamic-reconfiguration-memory ltcalpine2-lp9:/proc/device-tree/ibm,dynamic-reconfiguration-memory # ls ibm,associativity-lookup-arrays ibm,memory-flags-mask linux,phandle ibm,dynamic-memory ibm,memory-preservation-time name ibm,lmb-size ibm,phandle ltcalpine2-lp9:/proc/device-tree/ibm,dynamic-reconfiguration-memory # lsprop ibm,associativity-lookup-arrays ibm,associativity-lookup-arrays 00000004 00000004 00000000 00000000 00000000 00000000 00000000 00000000 00000001 00000001 00000000 00000003 00000006 00000006 00000000 00000003 00000007 00000007 ltcalpine2-lp9:/proc/device-tree/ibm,dynamic-reconfiguration-memory # lsprop ibm,dynamic-memory ibm,dynamic-memory 0000059e 00000000 20000000 80000002 00000000 00000000 00000008 00000000 30000000 80000003 00000000 00000000 00000008 00000000 40000000 80000004 00000000 00000000 00000008 00000000 50000000 80000005 00000000 00000000 00000008 00000000 60000000 80000006 00000000 00000000 00000008 00000000 70000000 80000007 00000000 00000000 00000008 00000000 80000000 80000008 00000000 00000000 00000008 00000000 90000000 80000009 00000000 00000000 00000008 00000000 a0000000 8000000a 00000000 00000000 00000008 00000000 b0000000 8000000b 00000000 00000000 00000008 00000000 c0000000 8000000c 00000000 00000000 00000008 00000000 d0000000 8000000d 00000000 00000000 00000008 00000000 e0000000 8000000e 00000000 00000000 00000008 00000000 f0000000 8000000f 00000000 00000000 00000008 00000001 00000000 80000010 00000000 00000000 00000008 00000001 10000000 80000011 00000000 00000000 00000008 00000001 20000000 80000012 00000000 00000000 00000008 00000001 30000000 80000013 00000000 00000000 00000008 00000001 40000000 80000014 00000000 00000000 00000008 00000001 50000000 80000015 00000000 00000000 00000008 00000001 60000000 80000016 00000000 00000000 00000008 00000001 70000000 80000017 00000000 00000000 00000008 00000001 80000000 80000018 00000000 00000000 00000008 00000001 90000000 80000019 00000000 00000000 00000008 00000001 a0000000 8000001a 00000000 00000000 00000008 00000001 b0000000 8000001b 00000000 00000000 00000008 00000001 c0000000 8000001c 00000000 00000000 00000008 00000001 d0000000 8000001d 00000000 00000000 00000008 00000001 e0000000 8000001e 00000000 00000000 00000008 00000001 f0000000 8000001f 00000000 00000000 00000008 00000002 00000000 80000020 00000000 00000000 00000008 00000002 10000000 80000021 00000000 00000000 00000008 00000002 20000000 80000022 00000000 00000000 00000008 00000002 30000000 80000023 00000000 00000000 00000008 00000002 40000000 80000024 00000000 00000000 00000008 00000002 50000000 80000025 00000000 00000000 00000008 00000002 60000000 80000026 00000000 00000000 00000008 00000002 70000000 80000027 00000000 00000000 00000008 00000002 80000000 80000028 00000000 00000000 00000008 00000002 90000000 80000029 00000000 00000000 00000008 00000002 a0000000 8000002a 00000000 00000000 00000008 00000002 b0000000 8000002b 00000000 00000000 00000008 00000002 c0000000 8000002c 00000000 00000000 00000008 00000002 d0000000 8000002d 00000000 00000000 00000008 00000002 e0000000 8000002e 00000000 00000000 00000008 00000002 f0000000 8000002f 00000000 00000000 00000008 00000003 00000000 80000030 00000000 00000000 00000008 00000003 10000000 80000031 00000000 00000000 00000008 00000003 20000000 80000032 00000000 00000000 00000008 00000003 30000000 80000033 00000000 00000000 00000008 00000003 40000000 80000034 00000000 00000000 00000008 00000003 50000000 80000035 00000000 00000000 00000008 00000003 60000000 80000036 00000000 00000000 00000008 00000003 70000000 80000037 00000000 00000000 00000008 00000003 80000000 80000038 00000000 00000000 00000008 00000003 90000000 80000039 00000000 00000000 00000008 00000003 a0000000 8000003a 00000000 00000000 00000008 00000003 b0000000 8000003b 00000000 00000000 00000008 00000003 c0000000 8000003c 00000000 00000000 00000008 00000003 d0000000 8000003d 00000000 00000000 00000008 00000003 e0000000 8000003e 00000000 00000000 00000008 00000003 f0000000 8000003f 00000000 00000000 00000008 00000004 00000000 80000040 00000000 00000000 00000008 00000004 10000000 80000041 00000000 00000000 00000008 00000004 20000000 80000042 00000000 00000000 00000008 00000004 30000000 80000043 00000000 00000000 00000008 00000004 40000000 80000044 00000000 00000000 00000008 00000004 50000000 80000045 00000000 00000000 00000008 00000004 60000000 80000046 00000000 00000000 00000008 00000004 70000000 80000047 00000000 00000000 00000008 00000004 80000000 80000048 00000000 00000000 00000008 00000004 90000000 80000049 00000000 00000000 00000008 00000004 a0000000 8000004a 00000000 00000000 00000008 00000004 b0000000 8000004b 00000000 00000000 00000008 00000004 c0000000 8000004c 00000000 00000000 00000008 00000004 d0000000 8000004d 00000000 00000000 00000008 00000004 e0000000 8000004e 00000000 00000000 00000008 00000004 f0000000 8000004f 00000000 00000000 00000008 00000005 00000000 80000050 00000000 00000000 00000008 00000005 10000000 80000051 00000000 00000001 00000008 00000005 20000000 80000052 00000000 00000001 00000008 00000005 30000000 80000053 00000000 00000001 00000008 00000005 40000000 80000054 00000000 00000001 00000008 00000005 50000000 80000055 00000000 00000001 00000008 00000005 60000000 80000056 00000000 00000001 00000008 00000005 70000000 80000057 00000000 00000001 00000008 00000005 80000000 80000058 00000000 00000001 00000008 00000005 90000000 80000059 00000000 00000001 00000008 00000005 a0000000 8000005a 00000000 00000001 00000008 00000005 b0000000 8000005b 00000000 00000001 00000008 00000005 c0000000 8000005c 00000000 00000002 00000008 00000005 d0000000 8000005d 00000000 00000002 00000008 00000005 e0000000 8000005e 00000000 00000002 00000008 00000005 f0000000 8000005f 00000000 00000002 00000008 00000006 00000000 80000060 00000000 00000002 00000008 00000006 10000000 80000061 00000000 00000002 00000008 00000006 20000000 80000062 00000000 00000002 00000008 00000006 30000000 80000063 00000000 00000002 00000008 00000006 40000000 80000064 00000000 00000002 00000008 00000006 50000000 80000065 00000000 00000002 00000008 00000006 60000000 80000066 00000000 00000002 00000008 00000006 70000000 80000067 00000000 00000002 00000008 00000006 80000000 80000068 00000000 00000002 00000008 00000006 90000000 80000069 00000000 00000002 00000008 00000006 a0000000 8000006a 00000000 00000002 00000008 00000006 b0000000 8000006b 00000000 00000002 00000008 00000006 c0000000 8000006c 00000000 00000002 00000008 00000006 d0000000 8000006d 00000000 00000002 00000008 00000006 e0000000 8000006e 00000000 00000002 00000008 00000006 f0000000 8000006f 00000000 00000002 00000008 00000007 00000000 80000070 00000000 00000002 00000008 00000007 10000000 80000071 00000000 00000002 00000008 00000007 20000000 80000072 00000000 00000002 00000008 00000007 30000000 80000073 00000000 00000002 00000008 00000007 40000000 80000074 00000000 00000002 00000008 00000007 50000000 80000075 00000000 00000002 00000008 00000007 60000000 80000076 00000000 00000002 00000008 00000007 70000000 80000077 00000000 00000002 00000008 00000007 80000000 80000078 00000000 00000002 00000008 00000007 90000000 80000079 00000000 00000002 00000008 00000007 a0000000 8000007a 00000000 00000002 00000008 00000007 b0000000 8000007b 00000000 00000002 00000008 00000007 c0000000 8000007c 00000000 00000002 00000008 00000007 d0000000 8000007d 00000000 00000002 00000008 00000007 e0000000 8000007e 00000000 00000002 00000008 00000007 f0000000 8000007f 00000000 00000002 00000008 00000008 00000000 80000080 00000000 00000002 00000008 00000008 10000000 80000081 00000000 00000002 00000008 00000008 20000000 80000082 00000000 00000002 00000008 00000008 30000000 80000083 00000000 00000002 00000008 00000008 40000000 80000084 00000000 00000002 00000008 00000008 50000000 80000085 00000000 00000002 00000008 00000008 60000000 80000086 00000000 00000002 00000008 00000008 70000000 80000087 00000000 00000002 00000008 00000008 80000000 80000088 00000000 00000002 00000008 00000008 90000000 80000089 00000000 00000002 00000008 00000008 a0000000 8000008a 00000000 00000002 00000008 00000008 b0000000 8000008b 00000000 00000002 00000008 00000008 c0000000 8000008c 00000000 00000002 00000008 00000008 d0000000 8000008d 00000000 00000002 00000008 00000008 e0000000 8000008e 00000000 00000002 00000008 00000008 f0000000 8000008f 00000000 00000002 00000008 00000009 00000000 80000090 00000000 00000002 00000008 00000009 10000000 80000091 00000000 00000002 00000008 00000009 20000000 80000092 00000000 00000002 00000008 00000009 30000000 80000093 00000000 00000002 00000008 00000009 40000000 80000094 00000000 00000002 00000008 00000009 50000000 80000095 00000000 00000002 00000008 00000009 60000000 80000096 00000000 00000002 00000008 00000009 70000000 80000097 00000000 00000002 00000008 00000009 80000000 80000098 00000000 00000002 00000008 00000009 90000000 80000099 00000000 00000002 00000008 00000009 a0000000 8000009a 00000000 00000002 00000008 00000009 b0000000 8000009b 00000000 00000002 00000008 00000009 c0000000 8000009c 00000000 00000002 00000008 00000009 d0000000 8000009d 00000000 00000002 00000008 00000009 e0000000 8000009e 00000000 00000002 00000008 00000009 f0000000 8000009f 00000000 00000002 00000008 0000000a 00000000 800000a0 00000000 00000002 00000008 0000000a 10000000 800000a1 00000000 00000002 00000008 0000000a 20000000 800000a2 00000000 00000002 00000008 0000000a 30000000 800000a3 00000000 00000002 00000008 0000000a 40000000 800000a4 00000000 00000002 00000008 0000000a 50000000 800000a5 00000000 00000002 00000008 0000000a 60000000 800000a6 00000000 00000002 00000008 0000000a 70000000 800000a7 00000000 00000002 00000008 0000000a 80000000 800000a8 00000000 00000002 00000008 0000000a 90000000 800000a9 00000000 00000002 00000008 0000000a a0000000 800000aa 00000000 00000002 00000008 0000000a b0000000 800000ab 00000000 00000002 00000008 0000000a c0000000 800000ac 00000000 00000002 00000008 0000000a d0000000 800000ad 00000000 00000002 00000008 0000000a e0000000 800000ae 00000000 00000002 00000008 0000000a f0000000 800000af 00000000 00000002 00000008 0000000b 00000000 800000b0 00000000 00000002 00000008 0000000b 10000000 800000b1 00000000 00000002 00000008 0000000b 20000000 800000b2 00000000 00000002 00000008 0000000b 30000000 800000b3 00000000 00000002 00000008 0000000b 40000000 800000b4 00000000 00000002 00000008 0000000b 50000000 800000b5 00000000 00000002 00000008 0000000b 60000000 800000b6 00000000 00000002 00000008 0000000b 70000000 800000b7 00000000 00000002 00000008 0000000b 80000000 800000b8 00000000 00000002 00000008 0000000b 90000000 800000b9 00000000 00000002 00000008 0000000b a0000000 800000ba 00000000 00000002 00000008 0000000b b0000000 800000bb 00000000 00000002 00000008 0000000b c0000000 800000bc 00000000 00000002 00000008 0000000b d0000000 800000bd 00000000 00000002 00000008 0000000b e0000000 800000be 00000000 00000002 00000008 0000000b f0000000 800000bf 00000000 00000002 00000008 0000000c 00000000 800000c0 00000000 00000002 00000008 0000000c 10000000 800000c1 00000000 00000002 00000008 0000000c 20000000 800000c2 00000000 00000002 00000008 0000000c 30000000 800000c3 00000000 00000002 00000008 0000000c 40000000 800000c4 00000000 00000002 00000008 0000000c 50000000 800000c5 00000000 00000002 00000008 0000000c 60000000 800000c6 00000000 00000002 00000008 0000000c 70000000 800000c7 00000000 00000002 00000008 0000000c 80000000 800000c8 00000000 00000002 00000008 0000000c 90000000 800000c9 00000000 00000002 00000008 0000000c a0000000 800000ca 00000000 00000002 00000008 0000000c b0000000 800000cb 00000000 00000002 00000008 0000000c c0000000 800000cc 00000000 00000002 00000008 0000000c d0000000 800000cd 00000000 00000002 00000008 0000000c e0000000 800000ce 00000000 00000002 00000008 0000000c f0000000 800000cf 00000000 00000002 00000008 0000000d 00000000 800000d0 00000000 00000002 00000008 0000000d 10000000 800000d1 00000000 00000002 00000008 0000000d 20000000 800000d2 00000000 00000002 00000008 0000000d 30000000 800000d3 00000000 00000002 00000008 0000000d 40000000 800000d4 00000000 00000002 00000008 0000000d 50000000 800000d5 00000000 00000002 00000008 0000000d 60000000 800000d6 00000000 00000002 00000008 0000000d 70000000 800000d7 00000000 00000002 00000008 0000000d 80000000 800000d8 00000000 00000002 00000008 0000000d 90000000 800000d9 00000000 00000002 00000008 0000000d a0000000 800000da 00000000 00000002 00000008 0000000d b0000000 800000db 00000000 00000002 00000008 0000000d c0000000 800000dc 00000000 00000002 00000008 0000000d d0000000 800000dd 00000000 00000002 00000008 0000000d e0000000 800000de 00000000 00000002 00000008 0000000d f0000000 800000df 00000000 00000002 00000008 0000000e 00000000 800000e0 00000000 00000002 00000008 0000000e 10000000 800000e1 00000000 00000002 00000008 0000000e 20000000 800000e2 00000000 00000002 00000008 0000000e 30000000 800000e3 00000000 00000002 00000008 0000000e 40000000 800000e4 00000000 00000002 00000008 0000000e 50000000 800000e5 00000000 00000002 00000008 0000000e 60000000 800000e6 00000000 00000002 00000008 0000000e 70000000 800000e7 00000000 00000002 00000008 0000000e 80000000 800000e8 00000000 00000002 00000008 0000000e 90000000 800000e9 00000000 00000002 00000008 0000000e a0000000 800000ea 00000000 00000002 00000008 0000000e b0000000 800000eb 00000000 00000002 00000008 0000000e c0000000 800000ec 00000000 00000002 00000008 0000000e d0000000 800000ed 00000000 00000002 00000008 0000000e e0000000 800000ee 00000000 00000002 00000008 0000000e f0000000 800000ef 00000000 00000002 00000008 0000000f 00000000 800000f0 00000000 00000002 00000008 0000000f 10000000 800000f1 00000000 00000002 00000008 0000000f 20000000 800000f2 00000000 00000002 00000008 0000000f 30000000 800000f3 00000000 00000002 00000008 0000000f 40000000 800000f4 00000000 00000002 00000008 0000000f 50000000 800000f5 00000000 00000002 00000008 0000000f 60000000 800000f6 00000000 00000002 00000008 0000000f 70000000 800000f7 00000000 00000002 00000008 0000000f 80000000 800000f8 00000000 00000002 00000008 0000000f 90000000 800000f9 00000000 00000002 00000008 0000000f a0000000 800000fa 00000000 00000002 00000008 0000000f b0000000 800000fb 00000000 00000002 00000008 0000000f c0000000 800000fc 00000000 00000002 00000008 0000000f d0000000 800000fd 00000000 00000002 00000008 0000000f e0000000 800000fe 00000000 00000002 00000008 0000000f f0000000 800000ff 00000000 00000002 00000008 00000010 00000000 80000100 00000000 00000002 00000008 00000010 10000000 80000101 00000000 00000002 00000008 00000010 20000000 80000102 00000000 00000002 00000008 00000010 30000000 80000103 00000000 00000002 00000008 00000010 40000000 80000104 00000000 00000002 00000008 00000010 50000000 80000105 00000000 00000002 00000008 00000010 60000000 80000106 00000000 00000002 00000008 00000010 70000000 80000107 00000000 00000002 00000008 00000010 80000000 80000108 00000000 00000002 00000008 00000010 90000000 80000109 00000000 00000002 00000008 00000010 a0000000 8000010a 00000000 00000002 00000008 00000010 b0000000 8000010b 00000000 00000002 00000008 00000010 c0000000 8000010c 00000000 00000002 00000008 00000010 d0000000 8000010d 00000000 00000002 00000008 00000010 e0000000 8000010e 00000000 00000002 00000008 00000010 f0000000 8000010f 00000000 00000002 00000008 00000011 00000000 80000110 00000000 00000002 00000008 00000011 10000000 80000111 00000000 00000002 00000008 00000011 20000000 80000112 00000000 00000002 00000008 00000011 30000000 80000113 00000000 00000002 00000008 00000011 40000000 80000114 00000000 00000002 00000008 00000011 50000000 80000115 00000000 00000002 00000008 00000011 60000000 80000116 00000000 00000002 00000008 00000011 70000000 80000117 00000000 00000002 00000008 00000011 80000000 80000118 00000000 00000002 00000008 00000011 90000000 80000119 00000000 00000002 00000008 00000011 a0000000 8000011a 00000000 00000002 00000008 00000011 b0000000 8000011b 00000000 00000002 00000008 00000011 c0000000 8000011c 00000000 00000002 00000008 00000011 d0000000 8000011d 00000000 00000002 00000008 00000011 e0000000 8000011e 00000000 00000002 00000008 00000011 f0000000 8000011f 00000000 00000002 00000008 00000012 00000000 80000120 00000000 00000002 00000008 00000012 10000000 80000121 00000000 00000002 00000008 00000012 20000000 80000122 00000000 00000002 00000008 00000012 30000000 80000123 00000000 00000002 00000008 00000012 40000000 80000124 00000000 00000002 00000008 00000012 50000000 80000125 00000000 00000002 00000008 00000012 60000000 80000126 00000000 00000002 00000008 00000012 70000000 80000127 00000000 00000002 00000008 00000012 80000000 80000128 00000000 00000002 00000008 00000012 90000000 80000129 00000000 00000002 00000008 00000012 a0000000 8000012a 00000000 00000002 00000008 00000012 b0000000 8000012b 00000000 00000002 00000008 00000012 c0000000 8000012c 00000000 00000002 00000008 00000012 d0000000 8000012d 00000000 00000002 00000008 00000012 e0000000 8000012e 00000000 00000002 00000008 00000012 f0000000 8000012f 00000000 00000002 00000008 00000013 00000000 80000130 00000000 00000002 00000008 00000013 10000000 80000131 00000000 00000002 00000008 00000013 20000000 80000132 00000000 00000002 00000008 00000013 30000000 80000133 00000000 00000002 00000008 00000013 40000000 80000134 00000000 00000002 00000008 00000013 50000000 80000135 00000000 00000002 00000008 00000013 60000000 80000136 00000000 00000002 00000008 00000013 70000000 80000137 00000000 00000002 00000008 00000013 80000000 80000138 00000000 00000002 00000008 00000013 90000000 80000139 00000000 00000002 00000008 00000013 a0000000 8000013a 00000000 00000002 00000008 00000013 b0000000 8000013b 00000000 00000002 00000008 00000013 c0000000 8000013c 00000000 00000002 00000008 00000013 d0000000 8000013d 00000000 00000002 00000008 00000013 e0000000 8000013e 00000000 00000002 00000008 00000013 f0000000 8000013f 00000000 00000002 00000008 00000014 00000000 80000140 00000000 00000002 00000008 00000014 10000000 80000141 00000000 00000002 00000008 00000014 20000000 80000142 00000000 00000002 00000008 00000014 30000000 80000143 00000000 00000002 00000008 00000014 40000000 80000144 00000000 00000002 00000008 00000014 50000000 80000145 00000000 00000002 00000008 00000014 60000000 80000146 00000000 00000002 00000008 00000014 70000000 80000147 00000000 00000002 00000008 00000014 80000000 80000148 00000000 00000002 00000008 00000014 90000000 80000149 00000000 00000002 00000008 00000014 a0000000 8000014a 00000000 00000002 00000008 00000014 b0000000 8000014b 00000000 00000002 00000008 00000014 c0000000 8000014c 00000000 00000002 00000008 00000014 d0000000 8000014d 00000000 00000002 00000008 00000014 e0000000 8000014e 00000000 00000002 00000008 00000014 f0000000 8000014f 00000000 00000002 00000008 00000015 00000000 80000150 00000000 00000002 00000008 00000015 10000000 80000151 00000000 00000002 00000008 00000015 20000000 80000152 00000000 00000002 00000008 00000015 30000000 80000153 00000000 00000002 00000008 00000015 40000000 80000154 00000000 00000002 00000008 00000015 50000000 80000155 00000000 00000002 00000008 00000015 60000000 80000156 00000000 00000002 00000008 00000015 70000000 80000157 00000000 00000002 00000008 00000015 80000000 80000158 00000000 00000002 00000008 00000015 90000000 80000159 00000000 00000002 00000008 00000015 a0000000 8000015a 00000000 00000002 00000008 00000015 b0000000 8000015b 00000000 00000002 00000008 00000015 c0000000 8000015c 00000000 00000002 00000008 00000015 d0000000 8000015d 00000000 00000002 00000008 00000015 e0000000 8000015e 00000000 00000002 00000008 00000015 f0000000 8000015f 00000000 00000002 00000008 00000016 00000000 80000160 00000000 00000002 00000008 00000016 10000000 80000161 00000000 00000002 00000008 00000016 20000000 80000162 00000000 00000002 00000008 00000016 30000000 80000163 00000000 00000002 00000008 00000016 40000000 80000164 00000000 00000002 00000008 00000016 50000000 80000165 00000000 00000002 00000008 00000016 60000000 80000166 00000000 00000002 00000008 00000016 70000000 80000167 00000000 00000002 00000008 00000016 80000000 80000168 00000000 00000002 00000008 00000016 90000000 80000169 00000000 00000002 00000008 00000016 a0000000 8000016a 00000000 00000002 00000008 00000016 b0000000 8000016b 00000000 00000002 00000008 00000016 c0000000 8000016c 00000000 00000002 00000008 00000016 d0000000 8000016d 00000000 00000002 00000008 00000016 e0000000 8000016e 00000000 00000002 00000008 00000016 f0000000 8000016f 00000000 00000002 00000008 00000017 00000000 80000170 00000000 00000002 00000008 00000017 10000000 80000171 00000000 00000002 00000008 00000017 20000000 80000172 00000000 00000002 00000008 00000017 30000000 80000173 00000000 00000002 00000008 00000017 40000000 80000174 00000000 00000002 00000008 00000017 50000000 80000175 00000000 00000002 00000008 00000017 60000000 80000176 00000000 00000002 00000008 00000017 70000000 80000177 00000000 00000002 00000008 00000017 80000000 80000178 00000000 00000002 00000008 00000017 90000000 80000179 00000000 00000002 00000008 00000017 a0000000 8000017a 00000000 00000002 00000008 00000017 b0000000 8000017b 00000000 00000002 00000008 00000017 c0000000 8000017c 00000000 00000002 00000008 00000017 d0000000 8000017d 00000000 00000002 00000008 00000017 e0000000 8000017e 00000000 00000002 00000008 00000017 f0000000 8000017f 00000000 00000002 00000008 00000018 00000000 80000180 00000000 00000002 00000008 00000018 10000000 80000181 00000000 00000002 00000008 00000018 20000000 80000182 00000000 00000002 00000008 00000018 30000000 80000183 00000000 00000002 00000008 00000018 40000000 80000184 00000000 00000002 00000008 00000018 50000000 80000185 00000000 00000002 00000008 00000018 60000000 80000186 00000000 00000002 00000008 00000018 70000000 80000187 00000000 00000002 00000008 00000018 80000000 80000188 00000000 00000002 00000008 00000018 90000000 80000189 00000000 00000002 00000008 00000018 a0000000 8000018a 00000000 00000002 00000008 00000018 b0000000 8000018b 00000000 00000002 00000008 00000018 c0000000 8000018c 00000000 00000002 00000008 00000018 d0000000 8000018d 00000000 00000002 00000008 00000018 e0000000 8000018e 00000000 00000002 00000008 00000018 f0000000 8000018f 00000000 00000002 00000008 00000019 00000000 80000190 00000000 00000002 00000008 00000019 10000000 80000191 00000000 00000002 00000008 00000019 20000000 80000192 00000000 00000002 00000008 00000019 30000000 80000193 00000000 00000002 00000008 00000019 40000000 80000194 00000000 00000002 00000008 00000019 50000000 80000195 00000000 00000002 00000008 00000019 60000000 80000196 00000000 00000002 00000008 00000019 70000000 80000197 00000000 00000002 00000008 00000019 80000000 80000198 00000000 00000002 00000008 00000019 90000000 80000199 00000000 00000002 00000008 00000019 a0000000 8000019a 00000000 00000002 00000008 00000019 b0000000 8000019b 00000000 00000002 00000008 00000019 c0000000 8000019c 00000000 00000002 00000008 00000019 d0000000 8000019d 00000000 00000002 00000008 00000019 e0000000 8000019e 00000000 00000002 00000008 00000019 f0000000 8000019f 00000000 00000002 00000008 0000001a 00000000 800001a0 00000000 00000002 00000008 0000001a 10000000 800001a1 00000000 00000002 00000008 0000001a 20000000 800001a2 00000000 00000002 00000008 0000001a 30000000 800001a3 00000000 00000002 00000008 0000001a 40000000 800001a4 00000000 00000002 00000008 0000001a 50000000 800001a5 00000000 00000002 00000008 0000001a 60000000 800001a6 00000000 00000002 00000008 0000001a 70000000 800001a7 00000000 00000002 00000008 0000001a 80000000 800001a8 00000000 00000002 00000008 0000001a 90000000 800001a9 00000000 00000002 00000008 0000001a a0000000 800001aa 00000000 00000002 00000008 0000001a b0000000 800001ab 00000000 00000002 00000008 0000001a c0000000 800001ac 00000000 00000002 00000008 0000001a d0000000 800001ad 00000000 00000002 00000008 0000001a e0000000 800001ae 00000000 00000002 00000008 0000001a f0000000 800001af 00000000 00000002 00000008 0000001b 00000000 800001b0 00000000 00000002 00000008 0000001b 10000000 800001b1 00000000 00000002 00000008 0000001b 20000000 800001b2 00000000 00000002 00000008 0000001b 30000000 800001b3 00000000 00000002 00000008 0000001b 40000000 800001b4 00000000 00000002 00000008 0000001b 50000000 800001b5 00000000 00000002 00000008 0000001b 60000000 800001b6 00000000 00000002 00000008 0000001b 70000000 800001b7 00000000 00000002 00000008 0000001b 80000000 800001b8 00000000 00000002 00000008 0000001b 90000000 800001b9 00000000 00000002 00000008 0000001b a0000000 800001ba 00000000 00000002 00000008 0000001b b0000000 800001bb 00000000 00000002 00000008 0000001b c0000000 800001bc 00000000 00000002 00000008 0000001b d0000000 800001bd 00000000 00000002 00000008 0000001b e0000000 800001be 00000000 00000002 00000008 0000001b f0000000 800001bf 00000000 00000002 00000008 0000001c 00000000 800001c0 00000000 00000002 00000008 0000001c 10000000 800001c1 00000000 00000002 00000008 0000001c 20000000 800001c2 00000000 00000002 00000008 0000001c 30000000 800001c3 00000000 00000002 00000008 0000001c 40000000 800001c4 00000000 00000002 00000008 0000001c 50000000 800001c5 00000000 00000002 00000008 0000001c 60000000 800001c6 00000000 00000002 00000008 0000001c 70000000 800001c7 00000000 00000002 00000008 0000001c 80000000 800001c8 00000000 00000002 00000008 0000001c 90000000 800001c9 00000000 00000002 00000008 0000001c a0000000 800001ca 00000000 00000002 00000008 0000001c b0000000 800001cb 00000000 00000002 00000008 0000001c c0000000 800001cc 00000000 00000002 00000008 0000001c d0000000 800001cd 00000000 00000002 00000008 0000001c e0000000 800001ce 00000000 00000002 00000008 0000001c f0000000 800001cf 00000000 00000002 00000008 0000001d 00000000 800001d0 00000000 00000002 00000008 0000001d 10000000 800001d1 00000000 00000002 00000008 0000001d 20000000 800001d2 00000000 00000002 00000008 0000001d 30000000 800001d3 00000000 00000002 00000008 0000001d 40000000 800001d4 00000000 00000002 00000008 0000001d 50000000 800001d5 00000000 00000002 00000008 0000001d 60000000 800001d6 00000000 00000002 00000008 0000001d 70000000 800001d7 00000000 00000002 00000008 0000001d 80000000 800001d8 00000000 00000002 00000008 0000001d 90000000 800001d9 00000000 00000002 00000008 0000001d a0000000 800001da 00000000 00000002 00000008 0000001d b0000000 800001db 00000000 00000002 00000008 0000001d c0000000 800001dc 00000000 00000002 00000008 0000001d d0000000 800001dd 00000000 00000002 00000008 0000001d e0000000 800001de 00000000 00000002 00000008 0000001d f0000000 800001df 00000000 00000002 00000008 0000001e 00000000 800001e0 00000000 00000002 00000008 0000001e 10000000 800001e1 00000000 00000002 00000008 0000001e 20000000 800001e2 00000000 00000002 00000008 0000001e 30000000 800001e3 00000000 00000002 00000008 0000001e 40000000 800001e4 00000000 00000002 00000008 0000001e 50000000 800001e5 00000000 00000002 00000008 0000001e 60000000 800001e6 00000000 00000002 00000008 0000001e 70000000 800001e7 00000000 00000002 00000008 0000001e 80000000 800001e8 00000000 00000002 00000008 0000001e 90000000 800001e9 00000000 00000002 00000008 0000001e a0000000 800001ea 00000000 00000002 00000008 0000001e b0000000 800001eb 00000000 00000002 00000008 0000001e c0000000 800001ec 00000000 00000002 00000008 0000001e d0000000 800001ed 00000000 00000002 00000008 0000001e e0000000 800001ee 00000000 00000002 00000008 0000001e f0000000 800001ef 00000000 00000002 00000008 0000001f 00000000 800001f0 00000000 00000002 00000008 0000001f 10000000 800001f1 00000000 00000002 00000008 0000001f 20000000 800001f2 00000000 00000002 00000008 0000001f 30000000 800001f3 00000000 00000002 00000008 0000001f 40000000 800001f4 00000000 00000002 00000008 0000001f 50000000 800001f5 00000000 00000002 00000008 0000001f 60000000 800001f6 00000000 00000002 00000008 0000001f 70000000 800001f7 00000000 00000002 00000008 0000001f 80000000 800001f8 00000000 00000002 00000008 0000001f 90000000 800001f9 00000000 00000002 00000008 0000001f a0000000 800001fa 00000000 00000002 00000008 0000001f b0000000 800001fb 00000000 00000002 00000008 0000001f c0000000 800001fc 00000000 00000002 00000008 0000001f d0000000 800001fd 00000000 00000002 00000008 0000001f e0000000 800001fe 00000000 00000002 00000008 0000001f f0000000 800001ff 00000000 00000002 00000008 00000020 00000000 80000200 00000000 00000002 00000008 00000020 10000000 80000201 00000000 00000002 00000008 00000020 20000000 80000202 00000000 00000002 00000008 00000020 30000000 80000203 00000000 00000002 00000008 00000020 40000000 80000204 00000000 00000002 00000008 00000020 50000000 80000205 00000000 00000002 00000008 00000020 60000000 80000206 00000000 00000002 00000008 00000020 70000000 80000207 00000000 00000002 00000008 00000020 80000000 80000208 00000000 00000002 00000008 00000020 90000000 80000209 00000000 00000002 00000008 00000020 a0000000 8000020a 00000000 00000002 00000008 00000020 b0000000 8000020b 00000000 00000002 00000008 00000020 c0000000 8000020c 00000000 00000002 00000008 00000020 d0000000 8000020d 00000000 00000002 00000008 00000020 e0000000 8000020e 00000000 00000002 00000008 00000020 f0000000 8000020f 00000000 00000002 00000008 00000021 00000000 80000210 00000000 00000002 00000008 00000021 10000000 80000211 00000000 00000002 00000008 00000021 20000000 80000212 00000000 00000002 00000008 00000021 30000000 80000213 00000000 00000002 00000008 00000021 40000000 80000214 00000000 00000002 00000008 00000021 50000000 80000215 00000000 00000002 00000008 00000021 60000000 80000216 00000000 00000002 00000008 00000021 70000000 80000217 00000000 00000002 00000008 00000021 80000000 80000218 00000000 00000002 00000008 00000021 90000000 80000219 00000000 00000002 00000008 00000021 a0000000 8000021a 00000000 00000002 00000008 00000021 b0000000 8000021b 00000000 00000002 00000008 00000021 c0000000 8000021c 00000000 00000002 00000008 00000021 d0000000 8000021d 00000000 00000002 00000008 00000021 e0000000 8000021e 00000000 00000002 00000008 00000021 f0000000 8000021f 00000000 00000002 00000008 00000022 00000000 80000220 00000000 00000002 00000008 00000022 10000000 80000221 00000000 00000002 00000008 00000022 20000000 80000222 00000000 00000002 00000008 00000022 30000000 80000223 00000000 00000002 00000008 00000022 40000000 80000224 00000000 00000002 00000008 00000022 50000000 80000225 00000000 00000002 00000008 00000022 60000000 80000226 00000000 00000002 00000008 00000022 70000000 80000227 00000000 00000002 00000008 00000022 80000000 80000228 00000000 00000002 00000008 00000022 90000000 80000229 00000000 00000002 00000008 00000022 a0000000 8000022a 00000000 00000002 00000008 00000022 b0000000 8000022b 00000000 00000002 00000008 00000022 c0000000 8000022c 00000000 00000002 00000008 00000022 d0000000 8000022d 00000000 00000002 00000008 00000022 e0000000 8000022e 00000000 00000002 00000008 00000022 f0000000 8000022f 00000000 00000002 00000008 00000023 00000000 80000230 00000000 00000002 00000008 00000023 10000000 80000231 00000000 00000002 00000008 00000023 20000000 80000232 00000000 00000002 00000008 00000023 30000000 80000233 00000000 00000002 00000008 00000023 40000000 80000234 00000000 00000002 00000008 00000023 50000000 80000235 00000000 00000002 00000008 00000023 60000000 80000236 00000000 00000002 00000008 00000023 70000000 80000237 00000000 00000002 00000008 00000023 80000000 80000238 00000000 00000002 00000008 00000023 90000000 80000239 00000000 00000002 00000008 00000023 a0000000 8000023a 00000000 00000002 00000008 00000023 b0000000 8000023b 00000000 00000002 00000008 00000023 c0000000 8000023c 00000000 00000002 00000008 00000023 d0000000 8000023d 00000000 00000002 00000008 00000023 e0000000 8000023e 00000000 00000002 00000008 00000023 f0000000 8000023f 00000000 00000002 00000008 00000024 00000000 80000240 00000000 00000002 00000008 00000024 10000000 80000241 00000000 00000002 00000008 00000024 20000000 80000242 00000000 00000002 00000008 00000024 30000000 80000243 00000000 00000002 00000008 00000024 40000000 80000244 00000000 00000002 00000008 00000024 50000000 80000245 00000000 00000002 00000008 00000024 60000000 80000246 00000000 00000002 00000008 00000024 70000000 80000247 00000000 00000002 00000008 00000024 80000000 80000248 00000000 00000002 00000008 00000024 90000000 80000249 00000000 00000002 00000008 00000024 a0000000 8000024a 00000000 00000002 00000008 00000024 b0000000 8000024b 00000000 00000002 00000008 00000024 c0000000 8000024c 00000000 00000002 00000008 00000024 d0000000 8000024d 00000000 00000002 00000008 00000024 e0000000 8000024e 00000000 00000002 00000008 00000024 f0000000 8000024f 00000000 00000002 00000008 00000025 00000000 80000250 00000000 00000002 00000008 00000025 10000000 80000251 00000000 00000002 00000008 00000025 20000000 80000252 00000000 00000002 00000008 00000025 30000000 80000253 00000000 00000002 00000008 00000025 40000000 80000254 00000000 00000002 00000008 00000025 50000000 80000255 00000000 00000002 00000008 00000025 60000000 80000256 00000000 00000002 00000008 00000025 70000000 80000257 00000000 00000002 00000008 00000025 80000000 80000258 00000000 00000002 00000008 00000025 90000000 80000259 00000000 00000002 00000008 00000025 a0000000 8000025a 00000000 00000002 00000008 00000025 b0000000 8000025b 00000000 00000002 00000008 00000025 c0000000 8000025c 00000000 00000002 00000008 00000025 d0000000 8000025d 00000000 00000002 00000008 00000025 e0000000 8000025e 00000000 00000002 00000008 00000025 f0000000 8000025f 00000000 00000002 00000008 00000026 00000000 80000260 00000000 00000002 00000008 00000026 10000000 80000261 00000000 00000002 00000008 00000026 20000000 80000262 00000000 00000002 00000008 00000026 30000000 80000263 00000000 00000002 00000008 00000026 40000000 80000264 00000000 00000002 00000008 00000026 50000000 80000265 00000000 00000002 00000008 00000026 60000000 80000266 00000000 00000002 00000008 00000026 70000000 80000267 00000000 00000002 00000008 00000026 80000000 80000268 00000000 00000002 00000008 00000026 90000000 80000269 00000000 00000002 00000008 00000026 a0000000 8000026a 00000000 00000002 00000008 00000026 b0000000 8000026b 00000000 00000003 00000008 00000026 c0000000 8000026c 00000000 00000003 00000008 00000026 d0000000 8000026d 00000000 00000003 00000008 00000026 e0000000 8000026e 00000000 00000003 00000008 00000026 f0000000 8000026f 00000000 00000003 00000008 00000027 00000000 80000270 00000000 00000003 00000008 00000027 10000000 80000271 00000000 00000003 00000008 00000027 20000000 80000272 00000000 00000003 00000008 00000027 30000000 80000273 00000000 00000003 00000008 00000027 40000000 80000274 00000000 00000003 00000008 00000027 50000000 80000275 00000000 00000003 00000008 00000027 60000000 80000276 00000000 00000003 00000008 00000027 70000000 80000277 00000000 00000003 00000008 00000027 80000000 80000278 00000000 00000003 00000008 00000027 90000000 80000279 00000000 00000003 00000008 00000027 a0000000 8000027a 00000000 00000003 00000008 00000027 b0000000 8000027b 00000000 00000003 00000008 00000027 c0000000 8000027c 00000000 00000003 00000008 00000027 d0000000 8000027d 00000000 00000003 00000008 00000027 e0000000 8000027e 00000000 00000003 00000008 00000027 f0000000 8000027f 00000000 00000003 00000008 00000028 00000000 80000280 00000000 00000003 00000008 00000028 10000000 80000281 00000000 00000003 00000008 00000028 20000000 80000282 00000000 00000003 00000008 00000028 30000000 80000283 00000000 00000003 00000008 00000028 40000000 80000284 00000000 00000003 00000008 00000028 50000000 80000285 00000000 00000003 00000008 00000028 60000000 80000286 00000000 00000003 00000008 00000028 70000000 80000287 00000000 00000003 00000008 00000028 80000000 80000288 00000000 00000003 00000008 00000028 90000000 80000289 00000000 00000003 00000008 00000028 a0000000 8000028a 00000000 00000003 00000008 00000028 b0000000 8000028b 00000000 00000003 00000008 00000028 c0000000 8000028c 00000000 00000003 00000008 00000028 d0000000 8000028d 00000000 00000003 00000008 00000028 e0000000 8000028e 00000000 00000003 00000008 00000028 f0000000 8000028f 00000000 00000003 00000008 00000029 00000000 80000290 00000000 00000003 00000008 00000029 10000000 80000291 00000000 00000003 00000008 00000029 20000000 80000292 00000000 00000003 00000008 00000029 30000000 80000293 00000000 00000003 00000008 00000029 40000000 80000294 00000000 00000003 00000008 00000029 50000000 80000295 00000000 00000003 00000008 00000029 60000000 80000296 00000000 00000003 00000008 00000029 70000000 80000297 00000000 00000003 00000008 00000029 80000000 80000298 00000000 00000003 00000008 00000029 90000000 80000299 00000000 00000003 00000008 00000029 a0000000 8000029a 00000000 00000003 00000008 00000029 b0000000 8000029b 00000000 00000003 00000008 00000029 c0000000 8000029c 00000000 00000003 00000008 00000029 d0000000 8000029d 00000000 00000003 00000008 00000029 e0000000 8000029e 00000000 00000003 00000008 00000029 f0000000 8000029f 00000000 00000003 00000008 0000002a 00000000 800002a0 00000000 00000003 00000008 0000002a 10000000 800002a1 00000000 00000003 00000008 0000002a 20000000 800002a2 00000000 00000003 00000008 0000002a 30000000 800002a3 00000000 00000003 00000008 0000002a 40000000 800002a4 00000000 00000003 00000008 0000002a 50000000 800002a5 00000000 00000003 00000008 0000002a 60000000 800002a6 00000000 00000003 00000008 0000002a 70000000 800002a7 00000000 00000003 00000008 0000002a 80000000 800002a8 00000000 00000003 00000008 0000002a 90000000 800002a9 00000000 00000003 00000008 0000002a a0000000 800002aa 00000000 00000003 00000008 0000002a b0000000 800002ab 00000000 00000003 00000008 0000002a c0000000 800002ac 00000000 00000003 00000008 0000002a d0000000 800002ad 00000000 00000003 00000008 0000002a e0000000 800002ae 00000000 00000003 00000008 0000002a f0000000 800002af 00000000 00000003 00000008 0000002b 00000000 800002b0 00000000 00000003 00000008 0000002b 10000000 800002b1 00000000 00000003 00000008 0000002b 20000000 800002b2 00000000 00000003 00000008 0000002b 30000000 800002b3 00000000 00000003 00000008 0000002b 40000000 800002b4 00000000 00000003 00000008 0000002b 50000000 800002b5 00000000 00000003 00000008 0000002b 60000000 800002b6 00000000 00000003 00000008 0000002b 70000000 800002b7 00000000 00000003 00000008 0000002b 80000000 800002b8 00000000 00000003 00000008 0000002b 90000000 800002b9 00000000 00000003 00000008 0000002b a0000000 800002ba 00000000 00000003 00000008 0000002b b0000000 800002bb 00000000 00000003 00000008 0000002b c0000000 800002bc 00000000 00000003 00000008 0000002b d0000000 800002bd 00000000 00000003 00000008 0000002b e0000000 800002be 00000000 00000003 00000008 0000002b f0000000 800002bf 00000000 00000003 00000008 0000002c 00000000 800002c0 00000000 00000003 00000008 0000002c 10000000 800002c1 00000000 00000003 00000008 0000002c 20000000 800002c2 00000000 00000003 00000008 0000002c 30000000 800002c3 00000000 00000003 00000008 0000002c 40000000 800002c4 00000000 00000003 00000008 0000002c 50000000 800002c5 00000000 00000003 00000008 0000002c 60000000 800002c6 00000000 00000003 00000008 0000002c 70000000 800002c7 00000000 00000003 00000008 0000002c 80000000 800002c8 00000000 00000003 00000008 0000002c 90000000 800002c9 00000000 00000003 00000008 0000002c a0000000 800002ca 00000000 00000003 00000008 0000002c b0000000 800002cb 00000000 00000003 00000008 0000002c c0000000 800002cc 00000000 00000003 00000008 0000002c d0000000 800002cd 00000000 00000003 00000008 0000002c e0000000 800002ce 00000000 00000003 00000008 0000002c f0000000 800002cf 00000000 00000003 00000008 0000002d 00000000 800002d0 00000000 00000003 00000008 0000002d 10000000 800002d1 00000000 00000003 00000008 0000002d 20000000 800002d2 00000000 00000003 00000008 0000002d 30000000 800002d3 00000000 00000003 00000008 0000002d 40000000 800002d4 00000000 00000003 00000008 0000002d 50000000 800002d5 00000000 00000003 00000008 0000002d 60000000 800002d6 00000000 00000003 00000008 0000002d 70000000 800002d7 00000000 00000003 00000008 0000002d 80000000 800002d8 00000000 00000003 00000008 0000002d 90000000 800002d9 00000000 00000003 00000008 0000002d a0000000 800002da 00000000 00000003 00000008 0000002d b0000000 800002db 00000000 00000003 00000008 0000002d c0000000 800002dc 00000000 00000003 00000008 0000002d d0000000 800002dd 00000000 00000003 00000008 0000002d e0000000 800002de 00000000 00000003 00000008 0000002d f0000000 800002df 00000000 00000003 00000008 0000002e 00000000 800002e0 00000000 00000003 00000008 0000002e 10000000 800002e1 00000000 00000003 00000008 0000002e 20000000 800002e2 00000000 00000003 00000008 0000002e 30000000 800002e3 00000000 00000003 00000008 0000002e 40000000 800002e4 00000000 00000003 00000008 0000002e 50000000 800002e5 00000000 00000003 00000008 0000002e 60000000 800002e6 00000000 00000003 00000008 0000002e 70000000 800002e7 00000000 00000003 00000008 0000002e 80000000 800002e8 00000000 00000003 00000008 0000002e 90000000 800002e9 00000000 00000003 00000008 0000002e a0000000 800002ea 00000000 00000003 00000008 0000002e b0000000 800002eb 00000000 00000003 00000008 0000002e c0000000 800002ec 00000000 00000003 00000008 0000002e d0000000 800002ed 00000000 00000003 00000008 0000002e e0000000 800002ee 00000000 00000003 00000008 0000002e f0000000 800002ef 00000000 00000003 00000008 0000002f 00000000 800002f0 00000000 00000003 00000008 0000002f 10000000 800002f1 00000000 00000003 00000008 0000002f 20000000 800002f2 00000000 00000003 00000008 0000002f 30000000 800002f3 00000000 00000003 00000008 0000002f 40000000 800002f4 00000000 00000003 00000008 0000002f 50000000 800002f5 00000000 00000003 00000008 0000002f 60000000 800002f6 00000000 00000003 00000008 0000002f 70000000 800002f7 00000000 00000003 00000008 0000002f 80000000 800002f8 00000000 00000003 00000008 0000002f 90000000 800002f9 00000000 00000003 00000008 0000002f a0000000 800002fa 00000000 00000003 00000008 0000002f b0000000 800002fb 00000000 00000003 00000008 0000002f c0000000 800002fc 00000000 00000003 00000008 0000002f d0000000 800002fd 00000000 00000003 00000008 0000002f e0000000 800002fe 00000000 00000003 00000008 0000002f f0000000 800002ff 00000000 00000003 00000008 00000030 00000000 80000300 00000000 00000003 00000008 00000030 10000000 80000301 00000000 00000003 00000008 00000030 20000000 80000302 00000000 00000003 00000008 00000030 30000000 80000303 00000000 00000003 00000008 00000030 40000000 80000304 00000000 00000003 00000008 00000030 50000000 80000305 00000000 00000003 00000008 00000030 60000000 80000306 00000000 00000003 00000008 00000030 70000000 80000307 00000000 00000003 00000008 00000030 80000000 80000308 00000000 00000003 00000008 00000030 90000000 80000309 00000000 00000003 00000008 00000030 a0000000 8000030a 00000000 00000003 00000008 00000030 b0000000 8000030b 00000000 00000003 00000008 00000030 c0000000 8000030c 00000000 00000003 00000008 00000030 d0000000 8000030d 00000000 00000003 00000008 00000030 e0000000 8000030e 00000000 00000003 00000008 00000030 f0000000 8000030f 00000000 00000003 00000008 00000031 00000000 80000310 00000000 00000003 00000008 00000031 10000000 80000311 00000000 00000003 00000008 00000031 20000000 80000312 00000000 00000003 00000008 00000031 30000000 80000313 00000000 00000003 00000008 00000031 40000000 80000314 00000000 00000003 00000008 00000031 50000000 80000315 00000000 00000003 00000008 00000031 60000000 80000316 00000000 00000003 00000008 00000031 70000000 80000317 00000000 00000003 00000008 00000031 80000000 80000318 00000000 00000003 00000008 00000031 90000000 80000319 00000000 00000003 00000008 00000031 a0000000 8000031a 00000000 00000003 00000008 00000031 b0000000 8000031b 00000000 00000003 00000008 00000031 c0000000 8000031c 00000000 00000003 00000008 00000031 d0000000 8000031d 00000000 00000003 00000008 00000031 e0000000 8000031e 00000000 00000003 00000008 00000031 f0000000 8000031f 00000000 00000003 00000008 00000032 00000000 80000320 00000000 00000003 00000008 00000032 10000000 80000321 00000000 00000003 00000008 00000032 20000000 80000322 00000000 00000003 00000008 00000032 30000000 80000323 00000000 00000003 00000008 00000032 40000000 80000324 00000000 00000003 00000008 00000032 50000000 80000325 00000000 00000003 00000008 00000032 60000000 80000326 00000000 00000003 00000008 00000032 70000000 80000327 00000000 00000003 00000008 00000032 80000000 80000328 00000000 00000003 00000008 00000032 90000000 80000329 00000000 00000003 00000008 00000032 a0000000 8000032a 00000000 00000003 00000008 00000032 b0000000 8000032b 00000000 00000003 00000008 00000032 c0000000 8000032c 00000000 00000003 00000008 00000032 d0000000 8000032d 00000000 00000003 00000008 00000032 e0000000 8000032e 00000000 00000003 00000008 00000032 f0000000 8000032f 00000000 00000003 00000008 00000033 00000000 80000330 00000000 00000003 00000008 00000033 10000000 80000331 00000000 00000003 00000008 00000033 20000000 80000332 00000000 00000003 00000008 00000033 30000000 80000333 00000000 00000003 00000008 00000033 40000000 80000334 00000000 00000003 00000008 00000033 50000000 80000335 00000000 00000003 00000008 00000033 60000000 80000336 00000000 00000003 00000008 00000033 70000000 80000337 00000000 00000003 00000008 00000033 80000000 80000338 00000000 00000003 00000008 00000033 90000000 80000339 00000000 00000003 00000008 00000033 a0000000 8000033a 00000000 00000003 00000008 00000033 b0000000 8000033b 00000000 00000003 00000008 00000033 c0000000 8000033c 00000000 00000003 00000008 00000033 d0000000 8000033d 00000000 00000003 00000008 00000033 e0000000 8000033e 00000000 00000003 00000008 00000033 f0000000 8000033f 00000000 00000003 00000008 00000034 00000000 80000340 00000000 00000003 00000008 00000034 10000000 80000341 00000000 00000003 00000008 00000034 20000000 80000342 00000000 00000003 00000008 00000034 30000000 80000343 00000000 00000003 00000008 00000034 40000000 80000344 00000000 00000003 00000008 00000034 50000000 80000345 00000000 00000003 00000008 00000034 60000000 80000346 00000000 00000003 00000008 00000034 70000000 80000347 00000000 00000003 00000008 00000034 80000000 80000348 00000000 00000003 00000008 00000034 90000000 80000349 00000000 00000003 00000008 00000034 a0000000 8000034a 00000000 00000003 00000008 00000034 b0000000 8000034b 00000000 00000003 00000008 00000034 c0000000 8000034c 00000000 00000003 00000008 00000034 d0000000 8000034d 00000000 00000003 00000008 00000034 e0000000 8000034e 00000000 00000003 00000008 00000034 f0000000 8000034f 00000000 00000003 00000008 00000035 00000000 80000350 00000000 00000003 00000008 00000035 10000000 80000351 00000000 00000003 00000008 00000035 20000000 80000352 00000000 00000003 00000008 00000035 30000000 80000353 00000000 00000003 00000008 00000035 40000000 80000354 00000000 00000003 00000008 00000035 50000000 80000355 00000000 00000003 00000008 00000035 60000000 80000356 00000000 00000003 00000008 00000035 70000000 80000357 00000000 00000003 00000008 00000035 80000000 80000358 00000000 00000003 00000008 00000035 90000000 80000359 00000000 00000003 00000008 00000035 a0000000 8000035a 00000000 00000003 00000008 00000035 b0000000 8000035b 00000000 00000003 00000008 00000035 c0000000 8000035c 00000000 00000003 00000008 00000035 d0000000 8000035d 00000000 00000003 00000008 00000035 e0000000 8000035e 00000000 00000003 00000008 00000035 f0000000 8000035f 00000000 00000003 00000008 00000036 00000000 80000360 00000000 00000003 00000008 00000036 10000000 80000361 00000000 00000003 00000008 00000036 20000000 80000362 00000000 00000003 00000008 00000036 30000000 80000363 00000000 00000003 00000008 00000036 40000000 80000364 00000000 00000003 00000008 00000036 50000000 80000365 00000000 00000003 00000008 00000036 60000000 80000366 00000000 00000003 00000008 00000036 70000000 80000367 00000000 00000003 00000008 00000036 80000000 80000368 00000000 00000003 00000008 00000036 90000000 80000369 00000000 00000003 00000008 00000036 a0000000 8000036a 00000000 00000003 00000008 00000036 b0000000 8000036b 00000000 00000003 00000008 00000036 c0000000 8000036c 00000000 00000003 00000008 00000036 d0000000 8000036d 00000000 00000003 00000008 00000036 e0000000 8000036e 00000000 00000003 00000008 00000036 f0000000 8000036f 00000000 00000003 00000008 00000037 00000000 80000370 00000000 00000003 00000008 00000037 10000000 80000371 00000000 00000003 00000008 00000037 20000000 80000372 00000000 00000003 00000008 00000037 30000000 80000373 00000000 00000003 00000008 00000037 40000000 80000374 00000000 00000003 00000008 00000037 50000000 80000375 00000000 00000003 00000008 00000037 60000000 80000376 00000000 00000003 00000008 00000037 70000000 80000377 00000000 00000003 00000008 00000037 80000000 80000378 00000000 00000003 00000008 00000037 90000000 80000379 00000000 00000003 00000008 00000037 a0000000 8000037a 00000000 00000003 00000008 00000037 b0000000 8000037b 00000000 00000003 00000008 00000037 c0000000 8000037c 00000000 00000003 00000008 00000037 d0000000 8000037d 00000000 00000003 00000008 00000037 e0000000 8000037e 00000000 00000003 00000008 00000037 f0000000 8000037f 00000000 00000003 00000008 00000038 00000000 80000380 00000000 00000003 00000008 00000038 10000000 80000381 00000000 00000003 00000008 00000038 20000000 80000382 00000000 00000003 00000008 00000038 30000000 80000383 00000000 00000003 00000008 00000038 40000000 80000384 00000000 00000003 00000008 00000038 50000000 80000385 00000000 00000003 00000008 00000038 60000000 80000386 00000000 00000003 00000008 00000038 70000000 80000387 00000000 00000003 00000008 00000038 80000000 80000388 00000000 00000003 00000008 00000038 90000000 80000389 00000000 00000003 00000008 00000038 a0000000 8000038a 00000000 00000003 00000008 00000038 b0000000 8000038b 00000000 00000003 00000008 00000038 c0000000 8000038c 00000000 00000003 00000008 00000038 d0000000 8000038d 00000000 00000003 00000008 00000038 e0000000 8000038e 00000000 00000003 00000008 00000038 f0000000 8000038f 00000000 00000003 00000008 00000039 00000000 80000390 00000000 00000003 00000008 00000039 10000000 80000391 00000000 00000003 00000008 00000039 20000000 80000392 00000000 00000003 00000008 00000039 30000000 80000393 00000000 00000003 00000008 00000039 40000000 80000394 00000000 00000003 00000008 00000039 50000000 80000395 00000000 00000003 00000008 00000039 60000000 80000396 00000000 00000003 00000008 00000039 70000000 80000397 00000000 00000003 00000008 00000039 80000000 80000398 00000000 00000003 00000008 00000039 90000000 80000399 00000000 00000003 00000008 00000039 a0000000 8000039a 00000000 00000003 00000008 00000039 b0000000 8000039b 00000000 00000003 00000008 00000039 c0000000 8000039c 00000000 00000003 00000008 00000039 d0000000 8000039d 00000000 00000003 00000008 00000039 e0000000 8000039e 00000000 00000003 00000008 00000039 f0000000 8000039f 00000000 00000003 00000008 0000003a 00000000 800003a0 00000000 00000003 00000008 0000003a 10000000 800003a1 00000000 00000003 00000008 0000003a 20000000 800003a2 00000000 00000003 00000008 0000003a 30000000 800003a3 00000000 00000003 00000008 0000003a 40000000 800003a4 00000000 00000003 00000008 0000003a 50000000 800003a5 00000000 00000003 00000008 0000003a 60000000 800003a6 00000000 00000003 00000008 0000003a 70000000 800003a7 00000000 00000003 00000008 0000003a 80000000 800003a8 00000000 00000003 00000008 0000003a 90000000 800003a9 00000000 00000003 00000008 0000003a a0000000 800003aa 00000000 00000003 00000008 0000003a b0000000 800003ab 00000000 00000003 00000008 0000003a c0000000 800003ac 00000000 00000003 00000008 0000003a d0000000 800003ad 00000000 00000003 00000008 0000003a e0000000 800003ae 00000000 00000003 00000008 0000003a f0000000 800003af 00000000 00000003 00000008 0000003b 00000000 800003b0 00000000 00000003 00000008 0000003b 10000000 800003b1 00000000 00000003 00000008 0000003b 20000000 800003b2 00000000 00000003 00000008 0000003b 30000000 800003b3 00000000 00000003 00000008 0000003b 40000000 800003b4 00000000 00000003 00000008 0000003b 50000000 800003b5 00000000 00000003 00000008 0000003b 60000000 800003b6 00000000 00000003 00000008 0000003b 70000000 800003b7 00000000 00000003 00000008 0000003b 80000000 800003b8 00000000 00000003 00000008 0000003b 90000000 800003b9 00000000 00000003 00000008 0000003b a0000000 800003ba 00000000 00000003 00000008 0000003b b0000000 800003bb 00000000 00000003 00000008 0000003b c0000000 800003bc 00000000 00000003 00000008 0000003b d0000000 800003bd 00000000 00000003 00000008 0000003b e0000000 800003be 00000000 00000003 00000008 0000003b f0000000 800003bf 00000000 00000003 00000008 0000003c 00000000 800003c0 00000000 ffffffff 00000000 0000003c 10000000 800003c1 00000000 ffffffff 00000000 0000003c 20000000 800003c2 00000000 ffffffff 00000000 0000003c 30000000 800003c3 00000000 ffffffff 00000000 0000003c 40000000 800003c4 00000000 ffffffff 00000000 0000003c 50000000 800003c5 00000000 ffffffff 00000000 0000003c 60000000 800003c6 00000000 ffffffff 00000000 0000003c 70000000 800003c7 00000000 ffffffff 00000000 0000003c 80000000 800003c8 00000000 ffffffff 00000000 0000003c 90000000 800003c9 00000000 ffffffff 00000000 0000003c a0000000 800003ca 00000000 ffffffff 00000000 0000003c b0000000 800003cb 00000000 ffffffff 00000000 0000003c c0000000 800003cc 00000000 ffffffff 00000000 0000003c d0000000 800003cd 00000000 ffffffff 00000000 0000003c e0000000 800003ce 00000000 ffffffff 00000000 0000003c f0000000 800003cf 00000000 ffffffff 00000000 0000003d 00000000 800003d0 00000000 ffffffff 00000000 0000003d 10000000 800003d1 00000000 ffffffff 00000000 0000003d 20000000 800003d2 00000000 ffffffff 00000000 0000003d 30000000 800003d3 00000000 ffffffff 00000000 0000003d 40000000 800003d4 00000000 ffffffff 00000000 0000003d 50000000 800003d5 00000000 ffffffff 00000000 0000003d 60000000 800003d6 00000000 ffffffff 00000000 0000003d 70000000 800003d7 00000000 ffffffff 00000000 0000003d 80000000 800003d8 00000000 ffffffff 00000000 0000003d 90000000 800003d9 00000000 ffffffff 00000000 0000003d a0000000 800003da 00000000 ffffffff 00000000 0000003d b0000000 800003db 00000000 ffffffff 00000000 0000003d c0000000 800003dc 00000000 ffffffff 00000000 0000003d d0000000 800003dd 00000000 ffffffff 00000000 0000003d e0000000 800003de 00000000 ffffffff 00000000 0000003d f0000000 800003df 00000000 ffffffff 00000000 0000003e 00000000 800003e0 00000000 ffffffff 00000000 0000003e 10000000 800003e1 00000000 ffffffff 00000000 0000003e 20000000 800003e2 00000000 ffffffff 00000000 0000003e 30000000 800003e3 00000000 ffffffff 00000000 0000003e 40000000 800003e4 00000000 ffffffff 00000000 0000003e 50000000 800003e5 00000000 ffffffff 00000000 0000003e 60000000 800003e6 00000000 ffffffff 00000000 0000003e 70000000 800003e7 00000000 ffffffff 00000000 0000003e 80000000 800003e8 00000000 ffffffff 00000000 0000003e 90000000 800003e9 00000000 ffffffff 00000000 0000003e a0000000 800003ea 00000000 ffffffff 00000000 0000003e b0000000 800003eb 00000000 ffffffff 00000000 0000003e c0000000 800003ec 00000000 ffffffff 00000000 0000003e d0000000 800003ed 00000000 ffffffff 00000000 0000003e e0000000 800003ee 00000000 ffffffff 00000000 0000003e f0000000 800003ef 00000000 ffffffff 00000000 0000003f 00000000 800003f0 00000000 ffffffff 00000000 0000003f 10000000 800003f1 00000000 ffffffff 00000000 0000003f 20000000 800003f2 00000000 ffffffff 00000000 0000003f 30000000 800003f3 00000000 ffffffff 00000000 0000003f 40000000 800003f4 00000000 ffffffff 00000000 0000003f 50000000 800003f5 00000000 ffffffff 00000000 0000003f 60000000 800003f6 00000000 ffffffff 00000000 0000003f 70000000 800003f7 00000000 ffffffff 00000000 0000003f 80000000 800003f8 00000000 ffffffff 00000000 0000003f 90000000 800003f9 00000000 ffffffff 00000000 0000003f a0000000 800003fa 00000000 ffffffff 00000000 0000003f b0000000 800003fb 00000000 ffffffff 00000000 0000003f c0000000 800003fc 00000000 ffffffff 00000000 0000003f d0000000 800003fd 00000000 ffffffff 00000000 0000003f e0000000 800003fe 00000000 ffffffff 00000000 0000003f f0000000 800003ff 00000000 ffffffff 00000000 00000040 00000000 80000400 00000000 ffffffff 00000000 00000040 10000000 80000401 00000000 ffffffff 00000000 00000040 20000000 80000402 00000000 ffffffff 00000000 00000040 30000000 80000403 00000000 ffffffff 00000000 00000040 40000000 80000404 00000000 ffffffff 00000000 00000040 50000000 80000405 00000000 ffffffff 00000000 00000040 60000000 80000406 00000000 ffffffff 00000000 00000040 70000000 80000407 00000000 ffffffff 00000000 00000040 80000000 80000408 00000000 ffffffff 00000000 00000040 90000000 80000409 00000000 ffffffff 00000000 00000040 a0000000 8000040a 00000000 ffffffff 00000000 00000040 b0000000 8000040b 00000000 ffffffff 00000000 00000040 c0000000 8000040c 00000000 ffffffff 00000000 00000040 d0000000 8000040d 00000000 ffffffff 00000000 00000040 e0000000 8000040e 00000000 ffffffff 00000000 00000040 f0000000 8000040f 00000000 ffffffff 00000000 00000041 00000000 80000410 00000000 ffffffff 00000000 00000041 10000000 80000411 00000000 ffffffff 00000000 00000041 20000000 80000412 00000000 ffffffff 00000000 00000041 30000000 80000413 00000000 ffffffff 00000000 00000041 40000000 80000414 00000000 ffffffff 00000000 00000041 50000000 80000415 00000000 ffffffff 00000000 00000041 60000000 80000416 00000000 ffffffff 00000000 00000041 70000000 80000417 00000000 ffffffff 00000000 00000041 80000000 80000418 00000000 ffffffff 00000000 00000041 90000000 80000419 00000000 ffffffff 00000000 00000041 a0000000 8000041a 00000000 ffffffff 00000000 00000041 b0000000 8000041b 00000000 ffffffff 00000000 00000041 c0000000 8000041c 00000000 ffffffff 00000000 00000041 d0000000 8000041d 00000000 ffffffff 00000000 00000041 e0000000 8000041e 00000000 ffffffff 00000000 00000041 f0000000 8000041f 00000000 ffffffff 00000000 00000042 00000000 80000420 00000000 ffffffff 00000000 00000042 10000000 80000421 00000000 ffffffff 00000000 00000042 20000000 80000422 00000000 ffffffff 00000000 00000042 30000000 80000423 00000000 ffffffff 00000000 00000042 40000000 80000424 00000000 ffffffff 00000000 00000042 50000000 80000425 00000000 ffffffff 00000000 00000042 60000000 80000426 00000000 ffffffff 00000000 00000042 70000000 80000427 00000000 ffffffff 00000000 00000042 80000000 80000428 00000000 ffffffff 00000000 00000042 90000000 80000429 00000000 ffffffff 00000000 00000042 a0000000 8000042a 00000000 ffffffff 00000000 00000042 b0000000 8000042b 00000000 ffffffff 00000000 00000042 c0000000 8000042c 00000000 ffffffff 00000000 00000042 d0000000 8000042d 00000000 ffffffff 00000000 00000042 e0000000 8000042e 00000000 ffffffff 00000000 00000042 f0000000 8000042f 00000000 ffffffff 00000000 00000043 00000000 80000430 00000000 ffffffff 00000000 00000043 10000000 80000431 00000000 ffffffff 00000000 00000043 20000000 80000432 00000000 ffffffff 00000000 00000043 30000000 80000433 00000000 ffffffff 00000000 00000043 40000000 80000434 00000000 ffffffff 00000000 00000043 50000000 80000435 00000000 ffffffff 00000000 00000043 60000000 80000436 00000000 ffffffff 00000000 00000043 70000000 80000437 00000000 ffffffff 00000000 00000043 80000000 80000438 00000000 ffffffff 00000000 00000043 90000000 80000439 00000000 ffffffff 00000000 00000043 a0000000 8000043a 00000000 ffffffff 00000000 00000043 b0000000 8000043b 00000000 ffffffff 00000000 00000043 c0000000 8000043c 00000000 ffffffff 00000000 00000043 d0000000 8000043d 00000000 ffffffff 00000000 00000043 e0000000 8000043e 00000000 ffffffff 00000000 00000043 f0000000 8000043f 00000000 ffffffff 00000000 00000044 00000000 80000440 00000000 ffffffff 00000000 00000044 10000000 80000441 00000000 ffffffff 00000000 00000044 20000000 80000442 00000000 ffffffff 00000000 00000044 30000000 80000443 00000000 ffffffff 00000000 00000044 40000000 80000444 00000000 ffffffff 00000000 00000044 50000000 80000445 00000000 ffffffff 00000000 00000044 60000000 80000446 00000000 ffffffff 00000000 00000044 70000000 80000447 00000000 ffffffff 00000000 00000044 80000000 80000448 00000000 ffffffff 00000000 00000044 90000000 80000449 00000000 ffffffff 00000000 00000044 a0000000 8000044a 00000000 ffffffff 00000000 00000044 b0000000 8000044b 00000000 ffffffff 00000000 00000044 c0000000 8000044c 00000000 ffffffff 00000000 00000044 d0000000 8000044d 00000000 ffffffff 00000000 00000044 e0000000 8000044e 00000000 ffffffff 00000000 00000044 f0000000 8000044f 00000000 ffffffff 00000000 00000045 00000000 80000450 00000000 ffffffff 00000000 00000045 10000000 80000451 00000000 ffffffff 00000000 00000045 20000000 80000452 00000000 ffffffff 00000000 00000045 30000000 80000453 00000000 ffffffff 00000000 00000045 40000000 80000454 00000000 ffffffff 00000000 00000045 50000000 80000455 00000000 ffffffff 00000000 00000045 60000000 80000456 00000000 ffffffff 00000000 00000045 70000000 80000457 00000000 ffffffff 00000000 00000045 80000000 80000458 00000000 ffffffff 00000000 00000045 90000000 80000459 00000000 ffffffff 00000000 00000045 a0000000 8000045a 00000000 ffffffff 00000000 00000045 b0000000 8000045b 00000000 ffffffff 00000000 00000045 c0000000 8000045c 00000000 ffffffff 00000000 00000045 d0000000 8000045d 00000000 ffffffff 00000000 00000045 e0000000 8000045e 00000000 ffffffff 00000000 00000045 f0000000 8000045f 00000000 ffffffff 00000000 00000046 00000000 80000460 00000000 ffffffff 00000000 00000046 10000000 80000461 00000000 ffffffff 00000000 00000046 20000000 80000462 00000000 ffffffff 00000000 00000046 30000000 80000463 00000000 ffffffff 00000000 00000046 40000000 80000464 00000000 ffffffff 00000000 00000046 50000000 80000465 00000000 ffffffff 00000000 00000046 60000000 80000466 00000000 ffffffff 00000000 00000046 70000000 80000467 00000000 ffffffff 00000000 00000046 80000000 80000468 00000000 ffffffff 00000000 00000046 90000000 80000469 00000000 ffffffff 00000000 00000046 a0000000 8000046a 00000000 ffffffff 00000000 00000046 b0000000 8000046b 00000000 ffffffff 00000000 00000046 c0000000 8000046c 00000000 ffffffff 00000000 00000046 d0000000 8000046d 00000000 ffffffff 00000000 00000046 e0000000 8000046e 00000000 ffffffff 00000000 00000046 f0000000 8000046f 00000000 ffffffff 00000000 00000047 00000000 80000470 00000000 ffffffff 00000000 00000047 10000000 80000471 00000000 ffffffff 00000000 00000047 20000000 80000472 00000000 ffffffff 00000000 00000047 30000000 80000473 00000000 ffffffff 00000000 00000047 40000000 80000474 00000000 ffffffff 00000000 00000047 50000000 80000475 00000000 ffffffff 00000000 00000047 60000000 80000476 00000000 ffffffff 00000000 00000047 70000000 80000477 00000000 ffffffff 00000000 00000047 80000000 80000478 00000000 ffffffff 00000000 00000047 90000000 80000479 00000000 ffffffff 00000000 00000047 a0000000 8000047a 00000000 ffffffff 00000000 00000047 b0000000 8000047b 00000000 ffffffff 00000000 00000047 c0000000 8000047c 00000000 ffffffff 00000000 00000047 d0000000 8000047d 00000000 ffffffff 00000000 00000047 e0000000 8000047e 00000000 ffffffff 00000000 00000047 f0000000 8000047f 00000000 ffffffff 00000000 00000048 00000000 80000480 00000000 ffffffff 00000000 00000048 10000000 80000481 00000000 ffffffff 00000000 00000048 20000000 80000482 00000000 ffffffff 00000000 00000048 30000000 80000483 00000000 ffffffff 00000000 00000048 40000000 80000484 00000000 ffffffff 00000000 00000048 50000000 80000485 00000000 ffffffff 00000000 00000048 60000000 80000486 00000000 ffffffff 00000000 00000048 70000000 80000487 00000000 ffffffff 00000000 00000048 80000000 80000488 00000000 ffffffff 00000000 00000048 90000000 80000489 00000000 ffffffff 00000000 00000048 a0000000 8000048a 00000000 ffffffff 00000000 00000048 b0000000 8000048b 00000000 ffffffff 00000000 00000048 c0000000 8000048c 00000000 ffffffff 00000000 00000048 d0000000 8000048d 00000000 ffffffff 00000000 00000048 e0000000 8000048e 00000000 ffffffff 00000000 00000048 f0000000 8000048f 00000000 ffffffff 00000000 00000049 00000000 80000490 00000000 ffffffff 00000000 00000049 10000000 80000491 00000000 ffffffff 00000000 00000049 20000000 80000492 00000000 ffffffff 00000000 00000049 30000000 80000493 00000000 ffffffff 00000000 00000049 40000000 80000494 00000000 ffffffff 00000000 00000049 50000000 80000495 00000000 ffffffff 00000000 00000049 60000000 80000496 00000000 ffffffff 00000000 00000049 70000000 80000497 00000000 ffffffff 00000000 00000049 80000000 80000498 00000000 ffffffff 00000000 00000049 90000000 80000499 00000000 ffffffff 00000000 00000049 a0000000 8000049a 00000000 ffffffff 00000000 00000049 b0000000 8000049b 00000000 ffffffff 00000000 00000049 c0000000 8000049c 00000000 ffffffff 00000000 00000049 d0000000 8000049d 00000000 ffffffff 00000000 00000049 e0000000 8000049e 00000000 ffffffff 00000000 00000049 f0000000 8000049f 00000000 ffffffff 00000000 0000004a 00000000 800004a0 00000000 ffffffff 00000000 0000004a 10000000 800004a1 00000000 ffffffff 00000000 0000004a 20000000 800004a2 00000000 ffffffff 00000000 0000004a 30000000 800004a3 00000000 ffffffff 00000000 0000004a 40000000 800004a4 00000000 ffffffff 00000000 0000004a 50000000 800004a5 00000000 ffffffff 00000000 0000004a 60000000 800004a6 00000000 ffffffff 00000000 0000004a 70000000 800004a7 00000000 ffffffff 00000000 0000004a 80000000 800004a8 00000000 ffffffff 00000000 0000004a 90000000 800004a9 00000000 ffffffff 00000000 0000004a a0000000 800004aa 00000000 ffffffff 00000000 0000004a b0000000 800004ab 00000000 ffffffff 00000000 0000004a c0000000 800004ac 00000000 ffffffff 00000000 0000004a d0000000 800004ad 00000000 ffffffff 00000000 0000004a e0000000 800004ae 00000000 ffffffff 00000000 0000004a f0000000 800004af 00000000 ffffffff 00000000 0000004b 00000000 800004b0 00000000 ffffffff 00000000 0000004b 10000000 800004b1 00000000 ffffffff 00000000 0000004b 20000000 800004b2 00000000 ffffffff 00000000 0000004b 30000000 800004b3 00000000 ffffffff 00000000 0000004b 40000000 800004b4 00000000 ffffffff 00000000 0000004b 50000000 800004b5 00000000 ffffffff 00000000 0000004b 60000000 800004b6 00000000 ffffffff 00000000 0000004b 70000000 800004b7 00000000 ffffffff 00000000 0000004b 80000000 800004b8 00000000 ffffffff 00000000 0000004b 90000000 800004b9 00000000 ffffffff 00000000 0000004b a0000000 800004ba 00000000 ffffffff 00000000 0000004b b0000000 800004bb 00000000 ffffffff 00000000 0000004b c0000000 800004bc 00000000 ffffffff 00000000 0000004b d0000000 800004bd 00000000 ffffffff 00000000 0000004b e0000000 800004be 00000000 ffffffff 00000000 0000004b f0000000 800004bf 00000000 ffffffff 00000000 0000004c 00000000 800004c0 00000000 ffffffff 00000000 0000004c 10000000 800004c1 00000000 ffffffff 00000000 0000004c 20000000 800004c2 00000000 ffffffff 00000000 0000004c 30000000 800004c3 00000000 ffffffff 00000000 0000004c 40000000 800004c4 00000000 ffffffff 00000000 0000004c 50000000 800004c5 00000000 ffffffff 00000000 0000004c 60000000 800004c6 00000000 ffffffff 00000000 0000004c 70000000 800004c7 00000000 ffffffff 00000000 0000004c 80000000 800004c8 00000000 ffffffff 00000000 0000004c 90000000 800004c9 00000000 ffffffff 00000000 0000004c a0000000 800004ca 00000000 ffffffff 00000000 0000004c b0000000 800004cb 00000000 ffffffff 00000000 0000004c c0000000 800004cc 00000000 ffffffff 00000000 0000004c d0000000 800004cd 00000000 ffffffff 00000000 0000004c e0000000 800004ce 00000000 ffffffff 00000000 0000004c f0000000 800004cf 00000000 ffffffff 00000000 0000004d 00000000 800004d0 00000000 ffffffff 00000000 0000004d 10000000 800004d1 00000000 ffffffff 00000000 0000004d 20000000 800004d2 00000000 ffffffff 00000000 0000004d 30000000 800004d3 00000000 ffffffff 00000000 0000004d 40000000 800004d4 00000000 ffffffff 00000000 0000004d 50000000 800004d5 00000000 ffffffff 00000000 0000004d 60000000 800004d6 00000000 ffffffff 00000000 0000004d 70000000 800004d7 00000000 ffffffff 00000000 0000004d 80000000 800004d8 00000000 ffffffff 00000000 0000004d 90000000 800004d9 00000000 ffffffff 00000000 0000004d a0000000 800004da 00000000 ffffffff 00000000 0000004d b0000000 800004db 00000000 ffffffff 00000000 0000004d c0000000 800004dc 00000000 ffffffff 00000000 0000004d d0000000 800004dd 00000000 ffffffff 00000000 0000004d e0000000 800004de 00000000 ffffffff 00000000 0000004d f0000000 800004df 00000000 ffffffff 00000000 0000004e 00000000 800004e0 00000000 ffffffff 00000000 0000004e 10000000 800004e1 00000000 ffffffff 00000000 0000004e 20000000 800004e2 00000000 ffffffff 00000000 0000004e 30000000 800004e3 00000000 ffffffff 00000000 0000004e 40000000 800004e4 00000000 ffffffff 00000000 0000004e 50000000 800004e5 00000000 ffffffff 00000000 0000004e 60000000 800004e6 00000000 ffffffff 00000000 0000004e 70000000 800004e7 00000000 ffffffff 00000000 0000004e 80000000 800004e8 00000000 ffffffff 00000000 0000004e 90000000 800004e9 00000000 ffffffff 00000000 0000004e a0000000 800004ea 00000000 ffffffff 00000000 0000004e b0000000 800004eb 00000000 ffffffff 00000000 0000004e c0000000 800004ec 00000000 ffffffff 00000000 0000004e d0000000 800004ed 00000000 ffffffff 00000000 0000004e e0000000 800004ee 00000000 ffffffff 00000000 0000004e f0000000 800004ef 00000000 ffffffff 00000000 0000004f 00000000 800004f0 00000000 ffffffff 00000000 0000004f 10000000 800004f1 00000000 ffffffff 00000000 0000004f 20000000 800004f2 00000000 ffffffff 00000000 0000004f 30000000 800004f3 00000000 ffffffff 00000000 0000004f 40000000 800004f4 00000000 ffffffff 00000000 0000004f 50000000 800004f5 00000000 ffffffff 00000000 0000004f 60000000 800004f6 00000000 ffffffff 00000000 0000004f 70000000 800004f7 00000000 ffffffff 00000000 0000004f 80000000 800004f8 00000000 ffffffff 00000000 0000004f 90000000 800004f9 00000000 ffffffff 00000000 0000004f a0000000 800004fa 00000000 ffffffff 00000000 0000004f b0000000 800004fb 00000000 ffffffff 00000000 0000004f c0000000 800004fc 00000000 ffffffff 00000000 0000004f d0000000 800004fd 00000000 ffffffff 00000000 0000004f e0000000 800004fe 00000000 ffffffff 00000000 0000004f f0000000 800004ff 00000000 ffffffff 00000000 00000050 00000000 80000500 00000000 ffffffff 00000000 00000050 10000000 80000501 00000000 ffffffff 00000000 00000050 20000000 80000502 00000000 ffffffff 00000000 00000050 30000000 80000503 00000000 ffffffff 00000000 00000050 40000000 80000504 00000000 ffffffff 00000000 00000050 50000000 80000505 00000000 ffffffff 00000000 00000050 60000000 80000506 00000000 ffffffff 00000000 00000050 70000000 80000507 00000000 ffffffff 00000000 00000050 80000000 80000508 00000000 ffffffff 00000000 00000050 90000000 80000509 00000000 ffffffff 00000000 00000050 a0000000 8000050a 00000000 ffffffff 00000000 00000050 b0000000 8000050b 00000000 ffffffff 00000000 00000050 c0000000 8000050c 00000000 ffffffff 00000000 00000050 d0000000 8000050d 00000000 ffffffff 00000000 00000050 e0000000 8000050e 00000000 ffffffff 00000000 00000050 f0000000 8000050f 00000000 ffffffff 00000000 00000051 00000000 80000510 00000000 ffffffff 00000000 00000051 10000000 80000511 00000000 ffffffff 00000000 00000051 20000000 80000512 00000000 ffffffff 00000000 00000051 30000000 80000513 00000000 ffffffff 00000000 00000051 40000000 80000514 00000000 ffffffff 00000000 00000051 50000000 80000515 00000000 ffffffff 00000000 00000051 60000000 80000516 00000000 ffffffff 00000000 00000051 70000000 80000517 00000000 ffffffff 00000000 00000051 80000000 80000518 00000000 ffffffff 00000000 00000051 90000000 80000519 00000000 ffffffff 00000000 00000051 a0000000 8000051a 00000000 ffffffff 00000000 00000051 b0000000 8000051b 00000000 ffffffff 00000000 00000051 c0000000 8000051c 00000000 ffffffff 00000000 00000051 d0000000 8000051d 00000000 ffffffff 00000000 00000051 e0000000 8000051e 00000000 ffffffff 00000000 00000051 f0000000 8000051f 00000000 ffffffff 00000000 00000052 00000000 80000520 00000000 ffffffff 00000000 00000052 10000000 80000521 00000000 ffffffff 00000000 00000052 20000000 80000522 00000000 ffffffff 00000000 00000052 30000000 80000523 00000000 ffffffff 00000000 00000052 40000000 80000524 00000000 ffffffff 00000000 00000052 50000000 80000525 00000000 ffffffff 00000000 00000052 60000000 80000526 00000000 ffffffff 00000000 00000052 70000000 80000527 00000000 ffffffff 00000000 00000052 80000000 80000528 00000000 ffffffff 00000000 00000052 90000000 80000529 00000000 ffffffff 00000000 00000052 a0000000 8000052a 00000000 ffffffff 00000000 00000052 b0000000 8000052b 00000000 ffffffff 00000000 00000052 c0000000 8000052c 00000000 ffffffff 00000000 00000052 d0000000 8000052d 00000000 ffffffff 00000000 00000052 e0000000 8000052e 00000000 ffffffff 00000000 00000052 f0000000 8000052f 00000000 ffffffff 00000000 00000053 00000000 80000530 00000000 ffffffff 00000000 00000053 10000000 80000531 00000000 ffffffff 00000000 00000053 20000000 80000532 00000000 ffffffff 00000000 00000053 30000000 80000533 00000000 ffffffff 00000000 00000053 40000000 80000534 00000000 ffffffff 00000000 00000053 50000000 80000535 00000000 ffffffff 00000000 00000053 60000000 80000536 00000000 ffffffff 00000000 00000053 70000000 80000537 00000000 ffffffff 00000000 00000053 80000000 80000538 00000000 ffffffff 00000000 00000053 90000000 80000539 00000000 ffffffff 00000000 00000053 a0000000 8000053a 00000000 ffffffff 00000000 00000053 b0000000 8000053b 00000000 ffffffff 00000000 00000053 c0000000 8000053c 00000000 ffffffff 00000000 00000053 d0000000 8000053d 00000000 ffffffff 00000000 00000053 e0000000 8000053e 00000000 ffffffff 00000000 00000053 f0000000 8000053f 00000000 ffffffff 00000000 00000054 00000000 80000540 00000000 ffffffff 00000000 00000054 10000000 80000541 00000000 ffffffff 00000000 00000054 20000000 80000542 00000000 ffffffff 00000000 00000054 30000000 80000543 00000000 ffffffff 00000000 00000054 40000000 80000544 00000000 ffffffff 00000000 00000054 50000000 80000545 00000000 ffffffff 00000000 00000054 60000000 80000546 00000000 ffffffff 00000000 00000054 70000000 80000547 00000000 ffffffff 00000000 00000054 80000000 80000548 00000000 ffffffff 00000000 00000054 90000000 80000549 00000000 ffffffff 00000000 00000054 a0000000 8000054a 00000000 ffffffff 00000000 00000054 b0000000 8000054b 00000000 ffffffff 00000000 00000054 c0000000 8000054c 00000000 ffffffff 00000000 00000054 d0000000 8000054d 00000000 ffffffff 00000000 00000054 e0000000 8000054e 00000000 ffffffff 00000000 00000054 f0000000 8000054f 00000000 ffffffff 00000000 00000055 00000000 80000550 00000000 ffffffff 00000000 00000055 10000000 80000551 00000000 ffffffff 00000000 00000055 20000000 80000552 00000000 ffffffff 00000000 00000055 30000000 80000553 00000000 ffffffff 00000000 00000055 40000000 80000554 00000000 ffffffff 00000000 00000055 50000000 80000555 00000000 ffffffff 00000000 00000055 60000000 80000556 00000000 ffffffff 00000000 00000055 70000000 80000557 00000000 ffffffff 00000000 00000055 80000000 80000558 00000000 ffffffff 00000000 00000055 90000000 80000559 00000000 ffffffff 00000000 00000055 a0000000 8000055a 00000000 ffffffff 00000000 00000055 b0000000 8000055b 00000000 ffffffff 00000000 00000055 c0000000 8000055c 00000000 ffffffff 00000000 00000055 d0000000 8000055d 00000000 ffffffff 00000000 00000055 e0000000 8000055e 00000000 ffffffff 00000000 00000055 f0000000 8000055f 00000000 ffffffff 00000000 00000056 00000000 80000560 00000000 ffffffff 00000000 00000056 10000000 80000561 00000000 ffffffff 00000000 00000056 20000000 80000562 00000000 ffffffff 00000000 00000056 30000000 80000563 00000000 ffffffff 00000000 00000056 40000000 80000564 00000000 ffffffff 00000000 00000056 50000000 80000565 00000000 ffffffff 00000000 00000056 60000000 80000566 00000000 ffffffff 00000000 00000056 70000000 80000567 00000000 ffffffff 00000000 00000056 80000000 80000568 00000000 ffffffff 00000000 00000056 90000000 80000569 00000000 ffffffff 00000000 00000056 a0000000 8000056a 00000000 ffffffff 00000000 00000056 b0000000 8000056b 00000000 ffffffff 00000000 00000056 c0000000 8000056c 00000000 ffffffff 00000000 00000056 d0000000 8000056d 00000000 ffffffff 00000000 00000056 e0000000 8000056e 00000000 ffffffff 00000000 00000056 f0000000 8000056f 00000000 ffffffff 00000000 00000057 00000000 80000570 00000000 ffffffff 00000000 00000057 10000000 80000571 00000000 ffffffff 00000000 00000057 20000000 80000572 00000000 ffffffff 00000000 00000057 30000000 80000573 00000000 ffffffff 00000000 00000057 40000000 80000574 00000000 ffffffff 00000000 00000057 50000000 80000575 00000000 ffffffff 00000000 00000057 60000000 80000576 00000000 ffffffff 00000000 00000057 70000000 80000577 00000000 ffffffff 00000000 00000057 80000000 80000578 00000000 ffffffff 00000000 00000057 90000000 80000579 00000000 ffffffff 00000000 00000057 a0000000 8000057a 00000000 ffffffff 00000000 00000057 b0000000 8000057b 00000000 ffffffff 00000000 00000057 c0000000 8000057c 00000000 ffffffff 00000000 00000057 d0000000 8000057d 00000000 ffffffff 00000000 00000057 e0000000 8000057e 00000000 ffffffff 00000000 00000057 f0000000 8000057f 00000000 ffffffff 00000000 00000058 00000000 80000580 00000000 ffffffff 00000000 00000058 10000000 80000581 00000000 ffffffff 00000000 00000058 20000000 80000582 00000000 ffffffff 00000000 00000058 30000000 80000583 00000000 ffffffff 00000000 00000058 40000000 80000584 00000000 ffffffff 00000000 00000058 50000000 80000585 00000000 ffffffff 00000000 00000058 60000000 80000586 00000000 ffffffff 00000000 00000058 70000000 80000587 00000000 ffffffff 00000000 00000058 80000000 80000588 00000000 ffffffff 00000000 00000058 90000000 80000589 00000000 ffffffff 00000000 00000058 a0000000 8000058a 00000000 ffffffff 00000000 00000058 b0000000 8000058b 00000000 ffffffff 00000000 00000058 c0000000 8000058c 00000000 ffffffff 00000000 00000058 d0000000 8000058d 00000000 ffffffff 00000000 00000058 e0000000 8000058e 00000000 ffffffff 00000000 00000058 f0000000 8000058f 00000000 ffffffff 00000000 00000059 00000000 80000590 00000000 ffffffff 00000000 00000059 10000000 80000591 00000000 ffffffff 00000000 00000059 20000000 80000592 00000000 ffffffff 00000000 00000059 30000000 80000593 00000000 ffffffff 00000000 00000059 40000000 80000594 00000000 ffffffff 00000000 00000059 50000000 80000595 00000000 ffffffff 00000000 00000059 60000000 80000596 00000000 ffffffff 00000000 00000059 70000000 80000597 00000000 ffffffff 00000000 00000059 80000000 80000598 00000000 ffffffff 00000000 00000059 90000000 80000599 00000000 ffffffff 00000000 00000059 a0000000 8000059a 00000000 ffffffff 00000000 00000059 b0000000 8000059b 00000000 ffffffff 00000000 00000059 c0000000 8000059c 00000000 ffffffff 00000000 00000059 d0000000 8000059d 00000000 ffffffff 00000000 00000059 e0000000 8000059e 00000000 ffffffff 00000000 00000059 f0000000 8000059f 00000000 ffffffff 00000000 ltcalpine2-lp9:/proc/device-tree/ibm,dynamic-reconfiguration-memory # ltcalpine2-lp9:/proc/device-tree/ibm,dynamic-reconfiguration-memory # cat ~/dmesg.log [ 0.000000] Linux version 4.4.63-2.1.wi91275.sp3.055a_27-default (geeko@buildhost) (gcc version 4.8.5 (SUSE Linux) ) #1 SMP Tue May 2 10:56:31 EDT 2017 (27e2849) [ 0.000000] Node 0 Memory: 0x0-0x510000000 [ 0.000000] Node 1 Memory: 0x510000000-0x5c0000000 [ 0.000000] Node 6 Memory: 0x5c0000000-0x26b0000000 [ 0.000000] Node 7 Memory: 0x26b0000000-0x3c00000000 [ 0.000000] numa: Initmem setup node 0 [mem 0x00000000-0x50fffffff] [ 0.000000] numa: NODE_DATA [mem 0x50fff6100-0x50fffffff] [ 0.000000] numa: Initmem setup node 1 [mem 0x510000000-0x5bfffffff] [ 0.000000] numa: NODE_DATA [mem 0x5bfff6100-0x5bfffffff] [ 0.000000] numa: Initmem setup node 6 [mem 0x5c0000000-0x26afffffff] [ 0.000000] numa: NODE_DATA [mem 0x26afff6100-0x26afffffff] [ 0.000000] numa: Initmem setup node 7 [mem 0x26b0000000-0x3bffffffff] [ 0.000000] numa: NODE_DATA [mem 0x3bffbe2100-0x3bffbebfff] [ 0.000000] Section 15357 and 15359 (node 7) have a circular dependency on usemap and pgdat allocations [ 0.000000] PPC64 nvram contains 15360 bytes [ 0.000000] Top of RAM: 0x3c00000000, Total RAM: 0x3c00000000 [ 0.000000] Memory hole size: 0MB [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000000000-0x0000003bffffffff] [ 0.000000] DMA32 empty [ 0.000000] Normal empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000000000-0x000000050fffffff] [ 0.000000] node 1: [mem 0x0000000510000000-0x00000005bfffffff] [ 0.000000] node 6: [mem 0x00000005c0000000-0x00000026afffffff] [ 0.000000] node 7: [mem 0x00000026b0000000-0x0000003bffffffff] [ 0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x000000050fffffff] [ 0.000000] On node 0 totalpages: 331776 [ 0.000000] DMA zone: 324 pages used for memmap [ 0.000000] DMA zone: 0 pages reserved [ 0.000000] DMA zone: 331776 pages, LIFO batch:1 [ 0.000000] Initmem setup node 1 [mem 0x0000000510000000-0x00000005bfffffff] [ 0.000000] On node 1 totalpages: 45056 [ 0.000000] DMA zone: 44 pages used for memmap [ 0.000000] DMA zone: 0 pages reserved [ 0.000000] DMA zone: 45056 pages, LIFO batch:1 [ 0.000000] Initmem setup node 6 [mem 0x00000005c0000000-0x00000026afffffff] [ 0.000000] On node 6 totalpages: 2158592 [ 0.000000] DMA zone: 2108 pages used for memmap [ 0.000000] DMA zone: 0 pages reserved [ 0.000000] DMA zone: 2158592 pages, LIFO batch:1 [ 0.000000] Initmem setup node 7 [mem 0x00000026b0000000-0x0000003bffffffff] [ 0.000000] On node 7 totalpages: 1396736 [ 0.000000] DMA zone: 1364 pages used for memmap [ 0.000000] DMA zone: 0 pages reserved [ 0.000000] DMA zone: 1396736 pages, LIFO batch:1 ltcalpine2-lp9:/proc/device-tree/ibm,dynamic-reconfiguration-memory #
Here is the information from 2 different kernels. I have not been able to retrieve the information matching yesterday's attachments, yet, as those dumps were acquired in April. Attached please find 2 dumps of similar material from kernels running with my current patches (Linux 4.4, Linux 4.12). On 06/07/2017 07:08 AM, Michael Ellerman wrote: > Michael Bringmann <mwb@linux.vnet.ibm.com> writes: > >> On 06/06/2017 04:48 AM, Michael Ellerman wrote: >>> Michael Bringmann <mwb@linux.vnet.ibm.com> writes: >>>> On 06/01/2017 04:36 AM, Michael Ellerman wrote: >>>>> Do you actually see mention of nodes 0 and 8 in the dmesg? >>>> >>>> When the 'numa.c' code is built with debug messages, and the system was >>>> given that configuration by pHyp, yes, I did. >>>> >>>>> What does it say? >>>> >>>> The debug message for each core thread would be something like, >>>> >>>> removing cpu 64 from node 0 >>>> adding cpu 64 to node 8 >>>> >>>> repeated for all 8 threads of the CPU, and usually with the messages >>>> for all of the CPUs coming out intermixed on the console/dmesg log. >>> >>> OK. I meant what do you see at boot. >> >> Here is an example with nodes 0,2,6,7, node 0 starts out empty: >> >> [ 0.000000] Initmem setup node 0 >> [ 0.000000] NODE_DATA [mem 0x3bff7d6300-0x3bff7dffff] >> [ 0.000000] NODE_DATA(0) on node 7 >> [ 0.000000] Initmem setup node 2 [mem 0x00000000-0x13ffffffff] >> [ 0.000000] NODE_DATA [mem 0x13ffff6300-0x13ffffffff] >> [ 0.000000] Initmem setup node 6 [mem 0x1400000000-0x34afffffff] >> [ 0.000000] NODE_DATA [mem 0x34afff6300-0x34afffffff] >> [ 0.000000] Initmem setup node 7 [mem 0x34b0000000-0x3bffffffff] >> [ 0.000000] NODE_DATA [mem 0x3bff7cc600-0x3bff7d62ff] >> >> [ 0.000000] Zone ranges: >> [ 0.000000] DMA [mem 0x0000000000000000-0x0000003bffffffff] >> [ 0.000000] DMA32 empty >> [ 0.000000] Normal empty >> [ 0.000000] Movable zone start for each node >> [ 0.000000] Early memory node ranges >> [ 0.000000] node 2: [mem 0x0000000000000000-0x00000013ffffffff] >> [ 0.000000] node 6: [mem 0x0000001400000000-0x00000034afffffff] >> [ 0.000000] node 7: [mem 0x00000034b0000000-0x0000003bffffffff] >> [ 0.000000] Could not find start_pfn for node 0 >> [ 0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x0000000000000000] >> [ 0.000000] Initmem setup node 2 [mem 0x0000000000000000-0x00000013ffffffff] >> [ 0.000000] Initmem setup node 6 [mem 0x0000001400000000-0x00000034afffffff] >> [ 0.000000] Initmem setup node 7 [mem 0x00000034b0000000-0x0000003bffffffff] >> [ 0.000000] percpu: Embedded 3 pages/cpu @c000003bf8000000 s155672 r0 d40936 u262144 >> [ 0.000000] Built 4 zonelists in Node order, mobility grouping on. Total pages: 3928320 >> >> and, >> >> [root@ltcalpine2-lp20 ~]# numactl --hardware >> available: 4 nodes (0,2,6-7) >> node 0 cpus: >> node 0 size: 0 MB >> node 0 free: 0 MB >> node 2 cpus: 16 17 18 19 20 21 22 23 32 33 34 35 36 37 38 39 56 57 58 59 60 61 62 63 >> node 2 size: 81792 MB >> node 2 free: 81033 MB >> node 6 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 24 25 26 27 28 29 30 31 40 41 42 43 44 45 46 47 >> node 6 size: 133743 MB >> node 6 free: 133097 MB >> node 7 cpus: 48 49 50 51 52 53 54 55 >> node 7 size: 29877 MB >> node 7 free: 29599 MB >> node distances: >> node 0 2 6 7 >> 0: 10 40 40 40 >> 2: 40 10 40 40 >> 6: 40 40 10 20 >> 7: 40 40 20 10 >> [root@ltcalpine2-lp20 ~]# > > What kernel is that running? > > And can you show me the full ibm,dynamic-memory and lookup-arrays > properties for that system? > > cheers > >
Michael Bringmann <mwb@linux.vnet.ibm.com> writes: > Here is the information from 2 different kernels. I have not been able to retrieve > the information matching yesterday's attachments, yet, as those dumps were > acquired in April. > > Attached please find 2 dumps of similar material from kernels running with my > current patches (Linux 4.4, Linux 4.12). OK thanks. I'd actually like to see the dmesg output from a kernel *without* your patches. Looking at the device tree properties: ltcalpine2-lp9:/proc/device-tree/ibm,dynamic-reconfiguration-memory # lsprop ibm,associativity-lookup-arrays ibm,associativity-lookup-arrays 00000004 = 4 arrays 00000004 = of 4 entries each 00000000 00000000 00000000 00000000 00000000 00000000 00000001 00000001 00000000 00000003 00000006 00000006 00000000 00000003 00000007 00000007 Which does tell us that nodes 0, 1, 6 and 7 exist. So your idea of looking at that and setting any node found in there online should work. My only worry is that behaviour appears to be completely undocumented in PAPR, ie. PAPR explicitly says that property only needs to contain values for LMBs present at boot. But possibly we can talk to the PowerVM/PAPR guys and have that changed so that it becomes something we can rely on. cheers
On a related note, we are discussing the addition of 2 new device-tree properties with Pete Heyrman and his fellows that should simplify the determination of the set of required nodes. * One property would provide the total/max number of nodes needed by the kernel on the current hardware. * A second property would provide the total/max number of nodes that the kernel could use on any system to which it could be migrated. These properties aren't available, yet, and it takes time to define new properties in the PAPR and have them implemented in pHyp and the kernel. As an intermediary step, the systems which are doing a lot of dynamic hot-add/hot-remove configuration could provide equivalent information to the PowerPC kernel with a command line parameter. The 'numa.c' code would then read this value and fill in the necessary entries in the 'node_possible_map'. Would you foresee any problems with using such a feature? Thanks. On 06/13/2017 05:45 AM, Michael Ellerman wrote: > Michael Bringmann <mwb@linux.vnet.ibm.com> writes: > >> Here is the information from 2 different kernels. I have not been able to retrieve >> the information matching yesterday's attachments, yet, as those dumps were >> acquired in April. >> >> Attached please find 2 dumps of similar material from kernels running with my >> current patches (Linux 4.4, Linux 4.12). > > OK thanks. > > I'd actually like to see the dmesg output from a kernel *without* your > patches. > > Looking at the device tree properties: > > ltcalpine2-lp9:/proc/device-tree/ibm,dynamic-reconfiguration-memory # lsprop ibm,associativity-lookup-arrays > ibm,associativity-lookup-arrays > 00000004 = 4 arrays > 00000004 = of 4 entries each > 00000000 00000000 00000000 00000000 > 00000000 00000000 00000001 00000001 > 00000000 00000003 00000006 00000006 > 00000000 00000003 00000007 00000007 > > > Which does tell us that nodes 0, 1, 6 and 7 exist. > > So your idea of looking at that and setting any node found in there > online should work. > > My only worry is that behaviour appears to be completely undocumented in > PAPR, ie. PAPR explicitly says that property only needs to contain > values for LMBs present at boot. > > But possibly we can talk to the PowerVM/PAPR guys and have that changed > so that it becomes something we can rely on. > > cheers > >
On Wed, Jun 14, 2017 at 8:21 AM, Michael Bringmann <mwb@linux.vnet.ibm.com> wrote: > On a related note, we are discussing the addition of 2 new device-tree > properties > with Pete Heyrman and his fellows that should simplify the determination > of the > set of required nodes. > > * One property would provide the total/max number of nodes needed by the > kernel > on the current hardware. > Yes, that would be nice to have > * A second property would provide the total/max number of nodes that the > kernel > could use on any system to which it could be migrated. > > Not sure about this one, are you suggesting more memory can be added depending on the migration target? > These properties aren't available, yet, and it takes time to define new > properties > in the PAPR and have them implemented in pHyp and the kernel. As an > intermediary > step, the systems which are doing a lot of dynamic hot-add/hot-remove > configuration > could provide equivalent information to the PowerPC kernel with a command > line > parameter. The 'numa.c' code would then read this value and fill in the > necessary > entries in the 'node_possible_map'. > > Would you foresee any problems with using such a feature? > Balbir Singh
On Wed, Jun 14, 2017 at 3:25 PM, Balbir Singh <bsingharora@gmail.com> wrote: > > > On Wed, Jun 14, 2017 at 8:21 AM, Michael Bringmann <mwb@linux.vnet.ibm.com> > wrote: >> >> On a related note, we are discussing the addition of 2 new device-tree >> properties >> with Pete Heyrman and his fellows that should simplify the determination >> of the >> set of required nodes. >> >> * One property would provide the total/max number of nodes needed by the >> kernel >> on the current hardware. > > Yes, that would be nice to have > >> >> * A second property would provide the total/max number of nodes that the >> kernel >> could use on any system to which it could be migrated. >> > Not sure about this one, are you suggesting more memory can be added depending on the migration target? > > >> >> These properties aren't available, yet, and it takes time to define new >> properties >> in the PAPR and have them implemented in pHyp and the kernel. As an >> intermediary >> step, the systems which are doing a lot of dynamic hot-add/hot-remove >> configuration >> could provide equivalent information to the PowerPC kernel with a command >> line >> parameter. The 'numa.c' code would then read this value and fill in the >> necessary >> entries in the 'node_possible_map'. >> >> Would you foresee any problems with using such a feature? > > Sorry my mailer goofed up, resending Balbir Singh
Hello: On 06/14/2017 12:27 AM, Balbir Singh wrote: > On Wed, Jun 14, 2017 at 3:25 PM, Balbir Singh <bsingharora@gmail.com> wrote: >> >> >> On Wed, Jun 14, 2017 at 8:21 AM, Michael Bringmann <mwb@linux.vnet.ibm.com> >> wrote: >>> >>> On a related note, we are discussing the addition of 2 new device-tree >>> properties >>> with Pete Heyrman and his fellows that should simplify the determination >>> of the >>> set of required nodes. >>> >>> * One property would provide the total/max number of nodes needed by the >>> kernel >>> on the current hardware. >> >> > > Yes, that would be nice to have > >> >>> >>> * A second property would provide the total/max number of nodes that the >>> kernel >>> could use on any system to which it could be migrated. >>> >> > > Not sure about this one, are you suggesting more memory can be added > depending on the migration target? We would use only one of these numbers to allocate nodes. I have only been on the periphery of the discussions, so I can not communicate the full reasoning as to why both measures would be needed. We would like to have the first number for node allocation/initialization, but if only the second value were provided, we would likely need to use it. >> >> >>> >>> These properties aren't available, yet, and it takes time to define new >>> properties >>> in the PAPR and have them implemented in pHyp and the kernel. As an >>> intermediary >>> step, the systems which are doing a lot of dynamic hot-add/hot-remove >>> configuration >>> could provide equivalent information to the PowerPC kernel with a command >>> line >>> parameter. The 'numa.c' code would then read this value and fill in the >>> necessary >>> entries in the 'node_possible_map'. >>> >>> Would you foresee any problems with using such a feature? >> >> > > Sorry my mailer goofed up, resending > > Balbir Singh > Thanks.
diff --git a/arch/powerpc/mm/numa.c b/arch/powerpc/mm/numa.c index 15c2dd5..3d58c1f 100644 --- a/arch/powerpc/mm/numa.c +++ b/arch/powerpc/mm/numa.c @@ -870,7 +870,7 @@ void __init dump_numa_cpu_topology(void) } /* Initialize NODE_DATA for a node on the local memory */ -static void __init setup_node_data(int nid, u64 start_pfn, u64 end_pfn) +static void setup_node_data(int nid, u64 start_pfn, u64 end_pfn) { u64 spanned_pages = end_pfn - start_pfn; const size_t nd_size = roundup(sizeof(pg_data_t), SMP_CACHE_BYTES); @@ -878,23 +878,41 @@ static void __init setup_node_data(int nid, u64 start_pfn, u64 end_pfn) void *nd; int tnid; - nd_pa = memblock_alloc_try_nid(nd_size, SMP_CACHE_BYTES, nid); - nd = __va(nd_pa); + if (!node_data[nid]) { + nd_pa = memblock_alloc_try_nid(nd_size, SMP_CACHE_BYTES, nid); + nd = __va(nd_pa); - /* report and initialize */ - pr_info(" NODE_DATA [mem %#010Lx-%#010Lx]\n", - nd_pa, nd_pa + nd_size - 1); - tnid = early_pfn_to_nid(nd_pa >> PAGE_SHIFT); - if (tnid != nid) - pr_info(" NODE_DATA(%d) on node %d\n", nid, tnid); + node_data[nid] = nd; + memset(NODE_DATA(nid), 0, sizeof(pg_data_t)); + NODE_DATA(nid)->node_id = nid; + + /* report and initialize */ + pr_info(" NODE_DATA [mem %#010Lx-%#010Lx]\n", + nd_pa, nd_pa + nd_size - 1); + tnid = early_pfn_to_nid(nd_pa >> PAGE_SHIFT); + if (tnid != nid) + pr_info(" NODE_DATA(%d) on node %d\n", nid, tnid); + } else { + nd_pa = (u64) node_data[nid]; + nd = __va(nd_pa); + } - node_data[nid] = nd; - memset(NODE_DATA(nid), 0, sizeof(pg_data_t)); - NODE_DATA(nid)->node_id = nid; NODE_DATA(nid)->node_start_pfn = start_pfn; NODE_DATA(nid)->node_spanned_pages = spanned_pages; } +static void setup_nodes(void) +{ + int i, l = 32 /* MAX_NUMNODES */; + + for (i = 0; i < l; i++) { + if (!node_possible(i)) { + setup_node_data(i, 0, 0); + node_set(i, node_possible_map); + } + } +} + void __init initmem_init(void) { int nid, cpu; @@ -914,6 +932,8 @@ void __init initmem_init(void) */ nodes_and(node_possible_map, node_possible_map, node_online_map); + setup_nodes(); + for_each_online_node(nid) { unsigned long start_pfn, end_pfn;
Removing or adding memory via the PowerPC hotplug interface shows anomalies in the association between memory and nodes. The code was updated to initialize more possible nodes to make them available to subsequent DLPAR hotplug-memory operations, even if they are not needed at boot time. Signed-off-by: Michael Bringmann <mwb@linux.vnet.ibm.com> --- arch/powerpc/mm/numa.c | 44 ++++++++++++++++++++++++++++++++------------ 1 file changed, 32 insertions(+), 12 deletions(-)