Elf Profiling and potential improvements

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Elf Profiling and potential improvements

Matthew Khouzam
Hello world,

I have been profiling CDT's elf parser, and have found some opportunities. Now please understand, I was torture testing it to make sure it works with Trace Compass for some upcoming feature ideas I was going to pitch, I'm just hoping to help more than myself with this effort.

I was testing an embedded system with a 1 GB executable. Why? in case someone else does that. Elf parsing works well and scaled OK. But the memory requirements are a bit high.

Basically the parser loads every symbol into an array then sorts it. See image below for a "smaller" test with 60k functions.


I have made a quick patch to allow iteration through the elf file instead of reading it all in one block. This is useful when you want to make a map of symbols to trace segments. 

I have questions about the API of that file though, it has many public/protected fields that kind of limit its modify-ability, what would be the CDT friendly way to improve it?

My endgame would be to have an iterator on the executable that gives all the symbols, not necessarily in order so an extender can use it with more degrees of freedom.

Thanks!

Matthew


Ericsson
 
Matthew Khouzam    
Software Developer
 
Ericsson
8275 Route Transcanadienne
H4S 0B6,Saint-Laurent, Quebec
Canada
Our commitment to Technology for Good and Diversity and Inclusion contributes to positive change.
Follow us on: Facebook LinkedIn Twitter

Legal entity:ERICSSON AB registration number 556056-6258, registered office in Stockholm
This communication is confidential. Our email terms: https://www.ericsson.com/en/legal/privacy/email-disclaimer

_______________________________________________
cdt-dev mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev
Reply | Threaded
Open this post in threaded view
|

Re: Elf Profiling and potential improvements

Alexander Fedorov
Hello Matthew,

Thanks a lot for sharing the result of your work with community!
The idea to switch to "iterable" style is definitely the right step forward. We already started a discussion regarding implementation details in Gerrit.

As for API - we have an opportunity to make it much better, as we plan a major version bump (to 10.0) in September.
I suggest to start working on the new API in the (not yet created) "org.eclipse.cdt.binutils" bundle - current implementation may depend on it but not vice-versa.

The idea of "org.eclipse.cdt.binutils" is to focus on utilities for binaries itself (.elf, .dwarf, others), without involving CDT Core, UI and so on. Ideally it should not have any dependency that requires OSGi running. Yes, to be CLI-friendly. Then we can add "org.eclipse.cdt.binutils.workspace" to integrate with IWorkspace, "org.eclipse.cdt.binutils.jface" - to provide preferences, wizards and editors, and so on.
Why? To keep our dependencies clear and to avoid the fate of the current CDT Core/CDT UI.

Regards,
AF

03.04.2020 6:28, Matthew Khouzam пишет:
Hello world,

I have been profiling CDT's elf parser, and have found some opportunities. Now please understand, I was torture testing it to make sure it works with Trace Compass for some upcoming feature ideas I was going to pitch, I'm just hoping to help more than myself with this effort.

I was testing an embedded system with a 1 GB executable. Why? in case someone else does that. Elf parsing works well and scaled OK. But the memory requirements are a bit high.

Basically the parser loads every symbol into an array then sorts it. See image below for a "smaller" test with 60k functions.

I have made a quick patch to allow iteration through the elf file instead of reading it all in one block. This is useful when you want to make a map of symbols to trace segments. 

I have questions about the API of that file though, it has many public/protected fields that kind of limit its modify-ability, what would be the CDT friendly way to improve it?

My endgame would be to have an iterator on the executable that gives all the symbols, not necessarily in order so an extender can use it with more degrees of freedom.

Thanks!

Matthew


Ericsson
 
Matthew Khouzam    
Software Developer
 
Ericsson
8275 Route Transcanadienne
H4S 0B6,Saint-Laurent, Quebec
Canada
Our commitment to Technology for Good and Diversity and Inclusion contributes to positive change.
Follow us on: Facebook LinkedIn Twitter

Legal entity:ERICSSON AB registration number 556056-6258, registered office in Stockholm
This communication is confidential. Our email terms: https://www.ericsson.com/en/legal/privacy/email-disclaimer

_______________________________________________
cdt-dev mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev


_______________________________________________
cdt-dev mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev
Reply | Threaded
Open this post in threaded view
|

Re: Elf Profiling and potential improvements

Jonah Graham
Hi Matthew,

This is great news that you are looking into it. I really like the idea of API refactoring being driven by actual usecases - TraceCompass is a great usecase!

> I have questions about the API of that file though, it has many public/protected fields that kind of limit its modify-ability, what would be the CDT friendly way to improve it?

To re-iterate the above and what Alexander said, we have the opportunity to move things forward here without being tied too much to existing API. While we want the big step forward we are having to tread carefully to make sure that any replacement we are requiring other consumers to move to have documented migration paths. There are dependencies throughout CDT and the external community on the existing parsers. What has happened in the past with these types of classes is they are copied-pasted and the old ones are deprecated*. The big challenge with that approach is actually deleting the deprecated ones - but now that we intend to announce deletions and delete API without bumping Major version - like Eclipse Platform does - at least we have a way forward.

* e.g. org.eclipse.cdt.utils.elf.AR, org.eclipse.cdt.utils.macho.MachOHelper, org.eclipse.cdt.utils.macho.MachO, org.eclipse.cdt.utils.coff.parser.PEParser, org.eclipse.cdt.utils.coff.PE and numerous others.

Jonah

~~~
Jonah Graham
Kichwa Coders
www.kichwacoders.com


On Fri, 3 Apr 2020 at 09:48, Alexander Fedorov <[hidden email]> wrote:
Hello Matthew,

Thanks a lot for sharing the result of your work with community!
The idea to switch to "iterable" style is definitely the right step forward. We already started a discussion regarding implementation details in Gerrit.

As for API - we have an opportunity to make it much better, as we plan a major version bump (to 10.0) in September.
I suggest to start working on the new API in the (not yet created) "org.eclipse.cdt.binutils" bundle - current implementation may depend on it but not vice-versa.

The idea of "org.eclipse.cdt.binutils" is to focus on utilities for binaries itself (.elf, .dwarf, others), without involving CDT Core, UI and so on. Ideally it should not have any dependency that requires OSGi running. Yes, to be CLI-friendly. Then we can add "org.eclipse.cdt.binutils.workspace" to integrate with IWorkspace, "org.eclipse.cdt.binutils.jface" - to provide preferences, wizards and editors, and so on.
Why? To keep our dependencies clear and to avoid the fate of the current CDT Core/CDT UI.

Regards,
AF

03.04.2020 6:28, Matthew Khouzam пишет:
Hello world,

I have been profiling CDT's elf parser, and have found some opportunities. Now please understand, I was torture testing it to make sure it works with Trace Compass for some upcoming feature ideas I was going to pitch, I'm just hoping to help more than myself with this effort.

I was testing an embedded system with a 1 GB executable. Why? in case someone else does that. Elf parsing works well and scaled OK. But the memory requirements are a bit high.

Basically the parser loads every symbol into an array then sorts it. See image below for a "smaller" test with 60k functions.

I have made a quick patch to allow iteration through the elf file instead of reading it all in one block. This is useful when you want to make a map of symbols to trace segments. 

I have questions about the API of that file though, it has many public/protected fields that kind of limit its modify-ability, what would be the CDT friendly way to improve it?

My endgame would be to have an iterator on the executable that gives all the symbols, not necessarily in order so an extender can use it with more degrees of freedom.

Thanks!

Matthew


Ericsson
 
Matthew Khouzam    
Software Developer
 
Ericsson
8275 Route Transcanadienne
H4S 0B6,Saint-Laurent, Quebec
Canada
Our commitment to Technology for Good and Diversity and Inclusion contributes to positive change.
Follow us on: Facebook LinkedIn Twitter

Legal entity:ERICSSON AB registration number 556056-6258, registered office in Stockholm
This communication is confidential. Our email terms: https://www.ericsson.com/en/legal/privacy/email-disclaimer

_______________________________________________
cdt-dev mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev

_______________________________________________
cdt-dev mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev

_______________________________________________
cdt-dev mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev
Reply | Threaded
Open this post in threaded view
|

Re: Elf Profiling and potential improvements

Marc-Andre Laperle-3
One thing to consider with deleting the old parsers is that it breaks existing projects and therefore existing workspaces, not just APIs. Eclipse platform always supports upgrading a workspace, or does the proper conversion under the hood.
If the old parser classes are dropped then the IDs have to be preserved (but associated to the new classes) or converted to the new ids. I don’t know of any hooks to do that kind of conversion in MBS automatically but it might not be difficult to make a basic one with those things hardcoded. If neither of these are done (keeping the old ids associated to the new class or converting the ids) then they can’t really be removed.

Marc-André

On Apr 3, 2020, at 11:57 AM, Jonah Graham <[hidden email]> wrote:

Hi Matthew,

This is great news that you are looking into it. I really like the idea of API refactoring being driven by actual usecases - TraceCompass is a great usecase!

> I have questions about the API of that file though, it has many public/protected fields that kind of limit its modify-ability, what would be the CDT friendly way to improve it?

To re-iterate the above and what Alexander said, we have the opportunity to move things forward here without being tied too much to existing API. While we want the big step forward we are having to tread carefully to make sure that any replacement we are requiring other consumers to move to have documented migration paths. There are dependencies throughout CDT and the external community on the existing parsers. What has happened in the past with these types of classes is they are copied-pasted and the old ones are deprecated*. The big challenge with that approach is actually deleting the deprecated ones - but now that we intend to announce deletions and delete API without bumping Major version - like Eclipse Platform does - at least we have a way forward.

* e.g. org.eclipse.cdt.utils.elf.AR, org.eclipse.cdt.utils.macho.MachOHelper, org.eclipse.cdt.utils.macho.MachO, org.eclipse.cdt.utils.coff.parser.PEParser, org.eclipse.cdt.utils.coff.PE and numerous others.

Jonah

~~~
Jonah Graham
Kichwa Coders
www.kichwacoders.com


On Fri, 3 Apr 2020 at 09:48, Alexander Fedorov <[hidden email]> wrote:
Hello Matthew,

Thanks a lot for sharing the result of your work with community!
The idea to switch to "iterable" style is definitely the right step forward. We already started a discussion regarding implementation details in Gerrit.

As for API - we have an opportunity to make it much better, as we plan a major version bump (to 10.0) in September.
I suggest to start working on the new API in the (not yet created) "org.eclipse.cdt.binutils" bundle - current implementation may depend on it but not vice-versa.

The idea of "org.eclipse.cdt.binutils" is to focus on utilities for binaries itself (.elf, .dwarf, others), without involving CDT Core, UI and so on. Ideally it should not have any dependency that requires OSGi running. Yes, to be CLI-friendly. Then we can add "org.eclipse.cdt.binutils.workspace" to integrate with IWorkspace, "org.eclipse.cdt.binutils.jface" - to provide preferences, wizards and editors, and so on.
Why? To keep our dependencies clear and to avoid the fate of the current CDT Core/CDT UI.

Regards,
AF

03.04.2020 6:28, Matthew Khouzam пишет:
Hello world,

I have been profiling CDT's elf parser, and have found some opportunities. Now please understand, I was torture testing it to make sure it works with Trace Compass for some upcoming feature ideas I was going to pitch, I'm just hoping to help more than myself with this effort.

I was testing an embedded system with a 1 GB executable. Why? in case someone else does that. Elf parsing works well and scaled OK. But the memory requirements are a bit high.

Basically the parser loads every symbol into an array then sorts it. See image below for a "smaller" test with 60k functions.

I have made a quick patch to allow iteration through the elf file instead of reading it all in one block. This is useful when you want to make a map of symbols to trace segments. 

I have questions about the API of that file though, it has many public/protected fields that kind of limit its modify-ability, what would be the CDT friendly way to improve it?

My endgame would be to have an iterator on the executable that gives all the symbols, not necessarily in order so an extender can use it with more degrees of freedom.

Thanks!

Matthew


Ericsson
 
Matthew Khouzam    
Software Developer
 
Ericsson
8275 Route Transcanadienne
H4S 0B6,Saint-Laurent, Quebec
Canada
Our commitment to Technology for Good and Diversity and Inclusion contributes to positive change.
Follow us on: Facebook LinkedIn Twitter

Legal entity:ERICSSON AB registration number 556056-6258, registered office in Stockholm
This communication is confidential. Our email terms: https://www.ericsson.com/en/legal/privacy/email-disclaimer

_______________________________________________
cdt-dev mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev

_______________________________________________
cdt-dev mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev
_______________________________________________
cdt-dev mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev


_______________________________________________
cdt-dev mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev
Reply | Threaded
Open this post in threaded view
|

Re: Elf Profiling and potential improvements

Jonah Graham
Thank you Marc-André - I will keep an eye out for plugin.xml changes that have the potential to break existing projects and workspaces!


~~~
Jonah Graham
Kichwa Coders
www.kichwacoders.com


On Wed, 8 Apr 2020 at 00:39, Marc-Andre Laperle <[hidden email]> wrote:
One thing to consider with deleting the old parsers is that it breaks existing projects and therefore existing workspaces, not just APIs. Eclipse platform always supports upgrading a workspace, or does the proper conversion under the hood.
If the old parser classes are dropped then the IDs have to be preserved (but associated to the new classes) or converted to the new ids. I don’t know of any hooks to do that kind of conversion in MBS automatically but it might not be difficult to make a basic one with those things hardcoded. If neither of these are done (keeping the old ids associated to the new class or converting the ids) then they can’t really be removed.

Marc-André

On Apr 3, 2020, at 11:57 AM, Jonah Graham <[hidden email]> wrote:

Hi Matthew,

This is great news that you are looking into it. I really like the idea of API refactoring being driven by actual usecases - TraceCompass is a great usecase!

> I have questions about the API of that file though, it has many public/protected fields that kind of limit its modify-ability, what would be the CDT friendly way to improve it?

To re-iterate the above and what Alexander said, we have the opportunity to move things forward here without being tied too much to existing API. While we want the big step forward we are having to tread carefully to make sure that any replacement we are requiring other consumers to move to have documented migration paths. There are dependencies throughout CDT and the external community on the existing parsers. What has happened in the past with these types of classes is they are copied-pasted and the old ones are deprecated*. The big challenge with that approach is actually deleting the deprecated ones - but now that we intend to announce deletions and delete API without bumping Major version - like Eclipse Platform does - at least we have a way forward.

* e.g. org.eclipse.cdt.utils.elf.AR, org.eclipse.cdt.utils.macho.MachOHelper, org.eclipse.cdt.utils.macho.MachO, org.eclipse.cdt.utils.coff.parser.PEParser, org.eclipse.cdt.utils.coff.PE and numerous others.

Jonah

~~~
Jonah Graham
Kichwa Coders
www.kichwacoders.com


On Fri, 3 Apr 2020 at 09:48, Alexander Fedorov <[hidden email]> wrote:
Hello Matthew,

Thanks a lot for sharing the result of your work with community!
The idea to switch to "iterable" style is definitely the right step forward. We already started a discussion regarding implementation details in Gerrit.

As for API - we have an opportunity to make it much better, as we plan a major version bump (to 10.0) in September.
I suggest to start working on the new API in the (not yet created) "org.eclipse.cdt.binutils" bundle - current implementation may depend on it but not vice-versa.

The idea of "org.eclipse.cdt.binutils" is to focus on utilities for binaries itself (.elf, .dwarf, others), without involving CDT Core, UI and so on. Ideally it should not have any dependency that requires OSGi running. Yes, to be CLI-friendly. Then we can add "org.eclipse.cdt.binutils.workspace" to integrate with IWorkspace, "org.eclipse.cdt.binutils.jface" - to provide preferences, wizards and editors, and so on.
Why? To keep our dependencies clear and to avoid the fate of the current CDT Core/CDT UI.

Regards,
AF

03.04.2020 6:28, Matthew Khouzam пишет:
Hello world,

I have been profiling CDT's elf parser, and have found some opportunities. Now please understand, I was torture testing it to make sure it works with Trace Compass for some upcoming feature ideas I was going to pitch, I'm just hoping to help more than myself with this effort.

I was testing an embedded system with a 1 GB executable. Why? in case someone else does that. Elf parsing works well and scaled OK. But the memory requirements are a bit high.

Basically the parser loads every symbol into an array then sorts it. See image below for a "smaller" test with 60k functions.

I have made a quick patch to allow iteration through the elf file instead of reading it all in one block. This is useful when you want to make a map of symbols to trace segments. 

I have questions about the API of that file though, it has many public/protected fields that kind of limit its modify-ability, what would be the CDT friendly way to improve it?

My endgame would be to have an iterator on the executable that gives all the symbols, not necessarily in order so an extender can use it with more degrees of freedom.

Thanks!

Matthew


Ericsson
 
Matthew Khouzam    
Software Developer
 
Ericsson
8275 Route Transcanadienne
H4S 0B6,Saint-Laurent, Quebec
Canada
Our commitment to Technology for Good and Diversity and Inclusion contributes to positive change.
Follow us on: Facebook LinkedIn Twitter

Legal entity:ERICSSON AB registration number 556056-6258, registered office in Stockholm
This communication is confidential. Our email terms: https://www.ericsson.com/en/legal/privacy/email-disclaimer

_______________________________________________
cdt-dev mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev

_______________________________________________
cdt-dev mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev
_______________________________________________
cdt-dev mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev

_______________________________________________
cdt-dev mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev

_______________________________________________
cdt-dev mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev
Reply | Threaded
Open this post in threaded view
|

Re: Elf Profiling and potential improvements

Alexander Fedorov
Migration support is a big and important topic. When we are talking about deletion of some API I assume that it should provide a way to migrate the user data. I don't believe that fully automated conversion is possible, in some cases it may require explicit action - with "Quick Fix" wizard for example. However, "aliasing" approach looks promising.

Regards,
AF

08.04.2020 17:21, Jonah Graham пишет:
Thank you Marc-André - I will keep an eye out for plugin.xml changes that have the potential to break existing projects and workspaces!


~~~
Jonah Graham
Kichwa Coders
www.kichwacoders.com


On Wed, 8 Apr 2020 at 00:39, Marc-Andre Laperle <[hidden email]> wrote:
One thing to consider with deleting the old parsers is that it breaks existing projects and therefore existing workspaces, not just APIs. Eclipse platform always supports upgrading a workspace, or does the proper conversion under the hood.
If the old parser classes are dropped then the IDs have to be preserved (but associated to the new classes) or converted to the new ids. I don’t know of any hooks to do that kind of conversion in MBS automatically but it might not be difficult to make a basic one with those things hardcoded. If neither of these are done (keeping the old ids associated to the new class or converting the ids) then they can’t really be removed.

Marc-André

On Apr 3, 2020, at 11:57 AM, Jonah Graham <[hidden email]> wrote:

Hi Matthew,

This is great news that you are looking into it. I really like the idea of API refactoring being driven by actual usecases - TraceCompass is a great usecase!

> I have questions about the API of that file though, it has many public/protected fields that kind of limit its modify-ability, what would be the CDT friendly way to improve it?

To re-iterate the above and what Alexander said, we have the opportunity to move things forward here without being tied too much to existing API. While we want the big step forward we are having to tread carefully to make sure that any replacement we are requiring other consumers to move to have documented migration paths. There are dependencies throughout CDT and the external community on the existing parsers. What has happened in the past with these types of classes is they are copied-pasted and the old ones are deprecated*. The big challenge with that approach is actually deleting the deprecated ones - but now that we intend to announce deletions and delete API without bumping Major version - like Eclipse Platform does - at least we have a way forward.

* e.g. org.eclipse.cdt.utils.elf.AR, org.eclipse.cdt.utils.macho.MachOHelper, org.eclipse.cdt.utils.macho.MachO, org.eclipse.cdt.utils.coff.parser.PEParser, org.eclipse.cdt.utils.coff.PE and numerous others.

Jonah

~~~
Jonah Graham
Kichwa Coders
www.kichwacoders.com


On Fri, 3 Apr 2020 at 09:48, Alexander Fedorov <[hidden email]> wrote:
Hello Matthew,

Thanks a lot for sharing the result of your work with community!
The idea to switch to "iterable" style is definitely the right step forward. We already started a discussion regarding implementation details in Gerrit.

As for API - we have an opportunity to make it much better, as we plan a major version bump (to 10.0) in September.
I suggest to start working on the new API in the (not yet created) "org.eclipse.cdt.binutils" bundle - current implementation may depend on it but not vice-versa.

The idea of "org.eclipse.cdt.binutils" is to focus on utilities for binaries itself (.elf, .dwarf, others), without involving CDT Core, UI and so on. Ideally it should not have any dependency that requires OSGi running. Yes, to be CLI-friendly. Then we can add "org.eclipse.cdt.binutils.workspace" to integrate with IWorkspace, "org.eclipse.cdt.binutils.jface" - to provide preferences, wizards and editors, and so on.
Why? To keep our dependencies clear and to avoid the fate of the current CDT Core/CDT UI.

Regards,
AF

03.04.2020 6:28, Matthew Khouzam пишет:
Hello world,

I have been profiling CDT's elf parser, and have found some opportunities. Now please understand, I was torture testing it to make sure it works with Trace Compass for some upcoming feature ideas I was going to pitch, I'm just hoping to help more than myself with this effort.

I was testing an embedded system with a 1 GB executable. Why? in case someone else does that. Elf parsing works well and scaled OK. But the memory requirements are a bit high.

Basically the parser loads every symbol into an array then sorts it. See image below for a "smaller" test with 60k functions.

I have made a quick patch to allow iteration through the elf file instead of reading it all in one block. This is useful when you want to make a map of symbols to trace segments. 

I have questions about the API of that file though, it has many public/protected fields that kind of limit its modify-ability, what would be the CDT friendly way to improve it?

My endgame would be to have an iterator on the executable that gives all the symbols, not necessarily in order so an extender can use it with more degrees of freedom.

Thanks!

Matthew


Ericsson
 
Matthew Khouzam    
Software Developer
 
Ericsson
8275 Route Transcanadienne
H4S 0B6,Saint-Laurent, Quebec
Canada
Our commitment to Technology for Good and Diversity and Inclusion contributes to positive change.
Follow us on: Facebook LinkedIn Twitter

Legal entity:ERICSSON AB registration number 556056-6258, registered office in Stockholm
This communication is confidential. Our email terms: https://www.ericsson.com/en/legal/privacy/email-disclaimer

_______________________________________________
cdt-dev mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev

_______________________________________________
cdt-dev mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev
_______________________________________________
cdt-dev mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev

_______________________________________________
cdt-dev mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev

_______________________________________________
cdt-dev mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev


_______________________________________________
cdt-dev mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev
Reply | Threaded
Open this post in threaded view
|

Re: Elf Profiling and potential improvements

Jan Baeyens

my 2cent

I'm not so fond of "quick Fix" wizards if they are part of the gui. The problem here is "When do you remove them". To avoid clutter that would be with the next release.
Due to this approach I once had to upgrade from (say) version 1 to (say)version 5, one by one. This because the quick fix from n-1 to n was removed in n+1.
So I had to install version 2,3 and 4 just to run the quick fixes so version 5 could quick fix version 4. This is not something I want to do again nor want to advice someone to do.

Best regards

Jantje

PS covid19 and Sloeber got me all caught up. Hoping to get back to future MBS soon

Op 8/04/2020 om 16:43 schreef Alexander Fedorov:
Migration support is a big and important topic. When we are talking about deletion of some API I assume that it should provide a way to migrate the user data. I don't believe that fully automated conversion is possible, in some cases it may require explicit action - with "Quick Fix" wizard for example. However, "aliasing" approach looks promising.

Regards,
AF

08.04.2020 17:21, Jonah Graham пишет:
Thank you Marc-André - I will keep an eye out for plugin.xml changes that have the potential to break existing projects and workspaces!


~~~
Jonah Graham
Kichwa Coders
www.kichwacoders.com


On Wed, 8 Apr 2020 at 00:39, Marc-Andre Laperle <[hidden email]> wrote:
One thing to consider with deleting the old parsers is that it breaks existing projects and therefore existing workspaces, not just APIs. Eclipse platform always supports upgrading a workspace, or does the proper conversion under the hood.
If the old parser classes are dropped then the IDs have to be preserved (but associated to the new classes) or converted to the new ids. I don’t know of any hooks to do that kind of conversion in MBS automatically but it might not be difficult to make a basic one with those things hardcoded. If neither of these are done (keeping the old ids associated to the new class or converting the ids) then they can’t really be removed.

Marc-André

On Apr 3, 2020, at 11:57 AM, Jonah Graham <[hidden email]> wrote:

Hi Matthew,

This is great news that you are looking into it. I really like the idea of API refactoring being driven by actual usecases - TraceCompass is a great usecase!

> I have questions about the API of that file though, it has many public/protected fields that kind of limit its modify-ability, what would be the CDT friendly way to improve it?

To re-iterate the above and what Alexander said, we have the opportunity to move things forward here without being tied too much to existing API. While we want the big step forward we are having to tread carefully to make sure that any replacement we are requiring other consumers to move to have documented migration paths. There are dependencies throughout CDT and the external community on the existing parsers. What has happened in the past with these types of classes is they are copied-pasted and the old ones are deprecated*. The big challenge with that approach is actually deleting the deprecated ones - but now that we intend to announce deletions and delete API without bumping Major version - like Eclipse Platform does - at least we have a way forward.

* e.g. org.eclipse.cdt.utils.elf.AR, org.eclipse.cdt.utils.macho.MachOHelper, org.eclipse.cdt.utils.macho.MachO, org.eclipse.cdt.utils.coff.parser.PEParser, org.eclipse.cdt.utils.coff.PE and numerous others.

Jonah

~~~
Jonah Graham
Kichwa Coders
www.kichwacoders.com


On Fri, 3 Apr 2020 at 09:48, Alexander Fedorov <[hidden email]> wrote:
Hello Matthew,

Thanks a lot for sharing the result of your work with community!
The idea to switch to "iterable" style is definitely the right step forward. We already started a discussion regarding implementation details in Gerrit.

As for API - we have an opportunity to make it much better, as we plan a major version bump (to 10.0) in September.
I suggest to start working on the new API in the (not yet created) "org.eclipse.cdt.binutils" bundle - current implementation may depend on it but not vice-versa.

The idea of "org.eclipse.cdt.binutils" is to focus on utilities for binaries itself (.elf, .dwarf, others), without involving CDT Core, UI and so on. Ideally it should not have any dependency that requires OSGi running. Yes, to be CLI-friendly. Then we can add "org.eclipse.cdt.binutils.workspace" to integrate with IWorkspace, "org.eclipse.cdt.binutils.jface" - to provide preferences, wizards and editors, and so on.
Why? To keep our dependencies clear and to avoid the fate of the current CDT Core/CDT UI.

Regards,
AF

03.04.2020 6:28, Matthew Khouzam пишет:
Hello world,

I have been profiling CDT's elf parser, and have found some opportunities. Now please understand, I was torture testing it to make sure it works with Trace Compass for some upcoming feature ideas I was going to pitch, I'm just hoping to help more than myself with this effort.

I was testing an embedded system with a 1 GB executable. Why? in case someone else does that. Elf parsing works well and scaled OK. But the memory requirements are a bit high.

Basically the parser loads every symbol into an array then sorts it. See image below for a "smaller" test with 60k functions.

I have made a quick patch to allow iteration through the elf file instead of reading it all in one block. This is useful when you want to make a map of symbols to trace segments. 

I have questions about the API of that file though, it has many public/protected fields that kind of limit its modify-ability, what would be the CDT friendly way to improve it?

My endgame would be to have an iterator on the executable that gives all the symbols, not necessarily in order so an extender can use it with more degrees of freedom.

Thanks!

Matthew


Ericsson
 
Matthew Khouzam    
Software Developer
 
Ericsson
8275 Route Transcanadienne
H4S 0B6,Saint-Laurent, Quebec
Canada
Our commitment to Technology for Good and Diversity and Inclusion contributes to positive change.
Follow us on: Facebook LinkedIn Twitter

Legal entity:ERICSSON AB registration number 556056-6258, registered office in Stockholm
This communication is confidential. Our email terms: https://www.ericsson.com/en/legal/privacy/email-disclaimer

_______________________________________________
cdt-dev mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev

_______________________________________________
cdt-dev mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev
_______________________________________________
cdt-dev mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev

_______________________________________________
cdt-dev mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev

_______________________________________________
cdt-dev mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev


_______________________________________________
cdt-dev mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev

_______________________________________________
cdt-dev mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev
Reply | Threaded
Open this post in threaded view
|

Re: Elf Profiling and potential improvements

Alexander Fedorov
"Quick Fix" is just a GUI part of functionality to enable a choice from several alternatives, and each "alternative" normally should be available as a "GUI-less" code to be executed in another context if needed.
Your scenario is possible in theory, if we will delete just introduced identifiers every version. But I don't believe that we ever will have ambitions and resources of that scale :)

Regards,
AF

08.04.2020 18:51, Jan Baeyens пишет:

my 2cent

I'm not so fond of "quick Fix" wizards if they are part of the gui. The problem here is "When do you remove them". To avoid clutter that would be with the next release.
Due to this approach I once had to upgrade from (say) version 1 to (say)version 5, one by one. This because the quick fix from n-1 to n was removed in n+1.
So I had to install version 2,3 and 4 just to run the quick fixes so version 5 could quick fix version 4. This is not something I want to do again nor want to advice someone to do.

Best regards

Jantje

PS covid19 and Sloeber got me all caught up. Hoping to get back to future MBS soon

Op 8/04/2020 om 16:43 schreef Alexander Fedorov:
Migration support is a big and important topic. When we are talking about deletion of some API I assume that it should provide a way to migrate the user data. I don't believe that fully automated conversion is possible, in some cases it may require explicit action - with "Quick Fix" wizard for example. However, "aliasing" approach looks promising.

Regards,
AF

08.04.2020 17:21, Jonah Graham пишет:
Thank you Marc-André - I will keep an eye out for plugin.xml changes that have the potential to break existing projects and workspaces!


~~~
Jonah Graham
Kichwa Coders
www.kichwacoders.com


On Wed, 8 Apr 2020 at 00:39, Marc-Andre Laperle <[hidden email]> wrote:
One thing to consider with deleting the old parsers is that it breaks existing projects and therefore existing workspaces, not just APIs. Eclipse platform always supports upgrading a workspace, or does the proper conversion under the hood.
If the old parser classes are dropped then the IDs have to be preserved (but associated to the new classes) or converted to the new ids. I don’t know of any hooks to do that kind of conversion in MBS automatically but it might not be difficult to make a basic one with those things hardcoded. If neither of these are done (keeping the old ids associated to the new class or converting the ids) then they can’t really be removed.

Marc-André

On Apr 3, 2020, at 11:57 AM, Jonah Graham <[hidden email]> wrote:

Hi Matthew,

This is great news that you are looking into it. I really like the idea of API refactoring being driven by actual usecases - TraceCompass is a great usecase!

> I have questions about the API of that file though, it has many public/protected fields that kind of limit its modify-ability, what would be the CDT friendly way to improve it?

To re-iterate the above and what Alexander said, we have the opportunity to move things forward here without being tied too much to existing API. While we want the big step forward we are having to tread carefully to make sure that any replacement we are requiring other consumers to move to have documented migration paths. There are dependencies throughout CDT and the external community on the existing parsers. What has happened in the past with these types of classes is they are copied-pasted and the old ones are deprecated*. The big challenge with that approach is actually deleting the deprecated ones - but now that we intend to announce deletions and delete API without bumping Major version - like Eclipse Platform does - at least we have a way forward.

* e.g. org.eclipse.cdt.utils.elf.AR, org.eclipse.cdt.utils.macho.MachOHelper, org.eclipse.cdt.utils.macho.MachO, org.eclipse.cdt.utils.coff.parser.PEParser, org.eclipse.cdt.utils.coff.PE and numerous others.

Jonah

~~~
Jonah Graham
Kichwa Coders
www.kichwacoders.com


On Fri, 3 Apr 2020 at 09:48, Alexander Fedorov <[hidden email]> wrote:
Hello Matthew,

Thanks a lot for sharing the result of your work with community!
The idea to switch to "iterable" style is definitely the right step forward. We already started a discussion regarding implementation details in Gerrit.

As for API - we have an opportunity to make it much better, as we plan a major version bump (to 10.0) in September.
I suggest to start working on the new API in the (not yet created) "org.eclipse.cdt.binutils" bundle - current implementation may depend on it but not vice-versa.

The idea of "org.eclipse.cdt.binutils" is to focus on utilities for binaries itself (.elf, .dwarf, others), without involving CDT Core, UI and so on. Ideally it should not have any dependency that requires OSGi running. Yes, to be CLI-friendly. Then we can add "org.eclipse.cdt.binutils.workspace" to integrate with IWorkspace, "org.eclipse.cdt.binutils.jface" - to provide preferences, wizards and editors, and so on.
Why? To keep our dependencies clear and to avoid the fate of the current CDT Core/CDT UI.

Regards,
AF

03.04.2020 6:28, Matthew Khouzam пишет:
Hello world,

I have been profiling CDT's elf parser, and have found some opportunities. Now please understand, I was torture testing it to make sure it works with Trace Compass for some upcoming feature ideas I was going to pitch, I'm just hoping to help more than myself with this effort.

I was testing an embedded system with a 1 GB executable. Why? in case someone else does that. Elf parsing works well and scaled OK. But the memory requirements are a bit high.

Basically the parser loads every symbol into an array then sorts it. See image below for a "smaller" test with 60k functions.

I have made a quick patch to allow iteration through the elf file instead of reading it all in one block. This is useful when you want to make a map of symbols to trace segments. 

I have questions about the API of that file though, it has many public/protected fields that kind of limit its modify-ability, what would be the CDT friendly way to improve it?

My endgame would be to have an iterator on the executable that gives all the symbols, not necessarily in order so an extender can use it with more degrees of freedom.

Thanks!

Matthew


Ericsson
 
Matthew Khouzam    
Software Developer
 
Ericsson
8275 Route Transcanadienne
H4S 0B6,Saint-Laurent, Quebec
Canada
Our commitment to Technology for Good and Diversity and Inclusion contributes to positive change.
Follow us on: Facebook LinkedIn Twitter

Legal entity:ERICSSON AB registration number 556056-6258, registered office in Stockholm
This communication is confidential. Our email terms: https://www.ericsson.com/en/legal/privacy/email-disclaimer

_______________________________________________
cdt-dev mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev

_______________________________________________
cdt-dev mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev
_______________________________________________
cdt-dev mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev

_______________________________________________
cdt-dev mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev

_______________________________________________
cdt-dev mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev


_______________________________________________
cdt-dev mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev

_______________________________________________
cdt-dev mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev


_______________________________________________
cdt-dev mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev
Reply | Threaded
Open this post in threaded view
|

Re: Elf Profiling and potential improvements

Jonah Graham
At least in this (binary parsers) case, I don't think there is any reason to remove or change existing identifiers.

Jonah 

On Wed., Apr. 8, 2020, 12:04 Alexander Fedorov, <[hidden email]> wrote:
"Quick Fix" is just a GUI part of functionality to enable a choice from several alternatives, and each "alternative" normally should be available as a "GUI-less" code to be executed in another context if needed.
Your scenario is possible in theory, if we will delete just introduced identifiers every version. But I don't believe that we ever will have ambitions and resources of that scale :)

Regards,
AF

08.04.2020 18:51, Jan Baeyens пишет:

my 2cent

I'm not so fond of "quick Fix" wizards if they are part of the gui. The problem here is "When do you remove them". To avoid clutter that would be with the next release.
Due to this approach I once had to upgrade from (say) version 1 to (say)version 5, one by one. This because the quick fix from n-1 to n was removed in n+1.
So I had to install version 2,3 and 4 just to run the quick fixes so version 5 could quick fix version 4. This is not something I want to do again nor want to advice someone to do.

Best regards

Jantje

PS covid19 and Sloeber got me all caught up. Hoping to get back to future MBS soon

Op 8/04/2020 om 16:43 schreef Alexander Fedorov:
Migration support is a big and important topic. When we are talking about deletion of some API I assume that it should provide a way to migrate the user data. I don't believe that fully automated conversion is possible, in some cases it may require explicit action - with "Quick Fix" wizard for example. However, "aliasing" approach looks promising.

Regards,
AF

08.04.2020 17:21, Jonah Graham пишет:
Thank you Marc-André - I will keep an eye out for plugin.xml changes that have the potential to break existing projects and workspaces!


~~~
Jonah Graham
Kichwa Coders
www.kichwacoders.com


On Wed, 8 Apr 2020 at 00:39, Marc-Andre Laperle <[hidden email]> wrote:
One thing to consider with deleting the old parsers is that it breaks existing projects and therefore existing workspaces, not just APIs. Eclipse platform always supports upgrading a workspace, or does the proper conversion under the hood.
If the old parser classes are dropped then the IDs have to be preserved (but associated to the new classes) or converted to the new ids. I don’t know of any hooks to do that kind of conversion in MBS automatically but it might not be difficult to make a basic one with those things hardcoded. If neither of these are done (keeping the old ids associated to the new class or converting the ids) then they can’t really be removed.

Marc-André

On Apr 3, 2020, at 11:57 AM, Jonah Graham <[hidden email]> wrote:

Hi Matthew,

This is great news that you are looking into it. I really like the idea of API refactoring being driven by actual usecases - TraceCompass is a great usecase!

> I have questions about the API of that file though, it has many public/protected fields that kind of limit its modify-ability, what would be the CDT friendly way to improve it?

To re-iterate the above and what Alexander said, we have the opportunity to move things forward here without being tied too much to existing API. While we want the big step forward we are having to tread carefully to make sure that any replacement we are requiring other consumers to move to have documented migration paths. There are dependencies throughout CDT and the external community on the existing parsers. What has happened in the past with these types of classes is they are copied-pasted and the old ones are deprecated*. The big challenge with that approach is actually deleting the deprecated ones - but now that we intend to announce deletions and delete API without bumping Major version - like Eclipse Platform does - at least we have a way forward.

* e.g. org.eclipse.cdt.utils.elf.AR, org.eclipse.cdt.utils.macho.MachOHelper, org.eclipse.cdt.utils.macho.MachO, org.eclipse.cdt.utils.coff.parser.PEParser, org.eclipse.cdt.utils.coff.PE and numerous others.

Jonah

~~~
Jonah Graham
Kichwa Coders
www.kichwacoders.com


On Fri, 3 Apr 2020 at 09:48, Alexander Fedorov <[hidden email]> wrote:
Hello Matthew,

Thanks a lot for sharing the result of your work with community!
The idea to switch to "iterable" style is definitely the right step forward. We already started a discussion regarding implementation details in Gerrit.

As for API - we have an opportunity to make it much better, as we plan a major version bump (to 10.0) in September.
I suggest to start working on the new API in the (not yet created) "org.eclipse.cdt.binutils" bundle - current implementation may depend on it but not vice-versa.

The idea of "org.eclipse.cdt.binutils" is to focus on utilities for binaries itself (.elf, .dwarf, others), without involving CDT Core, UI and so on. Ideally it should not have any dependency that requires OSGi running. Yes, to be CLI-friendly. Then we can add "org.eclipse.cdt.binutils.workspace" to integrate with IWorkspace, "org.eclipse.cdt.binutils.jface" - to provide preferences, wizards and editors, and so on.
Why? To keep our dependencies clear and to avoid the fate of the current CDT Core/CDT UI.

Regards,
AF

03.04.2020 6:28, Matthew Khouzam пишет:
Hello world,

I have been profiling CDT's elf parser, and have found some opportunities. Now please understand, I was torture testing it to make sure it works with Trace Compass for some upcoming feature ideas I was going to pitch, I'm just hoping to help more than myself with this effort.

I was testing an embedded system with a 1 GB executable. Why? in case someone else does that. Elf parsing works well and scaled OK. But the memory requirements are a bit high.

Basically the parser loads every symbol into an array then sorts it. See image below for a "smaller" test with 60k functions.

I have made a quick patch to allow iteration through the elf file instead of reading it all in one block. This is useful when you want to make a map of symbols to trace segments. 

I have questions about the API of that file though, it has many public/protected fields that kind of limit its modify-ability, what would be the CDT friendly way to improve it?

My endgame would be to have an iterator on the executable that gives all the symbols, not necessarily in order so an extender can use it with more degrees of freedom.

Thanks!

Matthew


Ericsson
 
Matthew Khouzam    
Software Developer
 
Ericsson
8275 Route Transcanadienne
H4S 0B6,Saint-Laurent, Quebec
Canada
Our commitment to Technology for Good and Diversity and Inclusion contributes to positive change.
Follow us on: Facebook LinkedIn Twitter

Legal entity:ERICSSON AB registration number 556056-6258, registered office in Stockholm
This communication is confidential. Our email terms: https://www.ericsson.com/en/legal/privacy/email-disclaimer

_______________________________________________
cdt-dev mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev

_______________________________________________
cdt-dev mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev
_______________________________________________
cdt-dev mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev

_______________________________________________
cdt-dev mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev

_______________________________________________
cdt-dev mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev


_______________________________________________
cdt-dev mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev

_______________________________________________
cdt-dev mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev

_______________________________________________
cdt-dev mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev

_______________________________________________
cdt-dev mailing list
[hidden email]
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdt-dev