Quantcast

etcd-provider - service keys to register service for discovery

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

etcd-provider - service keys to register service for discovery

Keimel, Christoph
Hi Scott,

I've been playing with the etcd discovery provider lately with the aim of replacing zeroconf for our escape game platform. It's very nice and really, really (!) fast. In the escape game deployment scenario the system will be running longer than 20 minutes, which is why I set TTL to 0 so the services will not be removed automatically. (The default TTL is 1200 seconds.)

This raises another issue though: As long as an application is shutdown gracefully the service key will be removed from etcd. But if you kill an application this is not the case. Unfortunately this is exactly what happens with the Raspberry Pi gateways on site. The guys power up the pi at the morning and at the evening they just kill the power to shut it down. The effect is that the keys will stay in etcd for ever. Since the keys are generated randomly for every session this leads to duplicates of services in etcd.

I had a look at EtcdDiscoveryContainer and I was wondering if we could use a key to register the service with etcd that doesn't change between session? What do you think?

Regards,
Christoph
--
Christoph Keimel

EM-SOFTWARE GmbH
Oskar-Messter-Str. 29
85737 Ismaning


Tel.: 089-99 65 47-26 Fax.: 089-99 65 47-99
Web: www.emsw.de E-Mail: [hidden email]

Geschäftsführer: Dipl.-Inf. (FH) Georg Engl
UST-Id-Nr.: DE 131 175 644, HRB 80271 München

_______________________________________________
ecf-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/ecf-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: etcd-provider - service keys to register service for discovery

Scott Lewis-2
On 11/25/2016 12:20 PM, Keimel, Christoph wrote:
> Hi Scott,
>
> I've been playing with the etcd discovery provider lately with the aim of replacing zeroconf for our escape game platform. It's very nice and really, really (!) fast. In the escape game deployment scenario the system will be running longer than 20 minutes, which is why I set TTL to 0 so the services will not be removed automatically. (The default TTL is 1200 seconds.)
>
> This raises another issue though: As long as an application is shutdown gracefully the service key will be removed from etcd. But if you kill an application this is not the case. Unfortunately this is exactly what happens with the Raspberry Pi gateways on site. The guys power up the pi at the morning and at the evening they just kill the power to shut it down. The effect is that the keys will stay in etcd for ever. Since the keys are generated randomly for every session this leads to duplicates of services in etcd.
>
> I had a look at EtcdDiscoveryContainer and I was wondering if we could use a key to register the service with etcd that doesn't change between session? What do you think?

Yeah, I imagine that you could publish/advertise a service with a
consistent key.   I can't remember if this will require any changes to
the etcd provider (e.g. to allow the key to be explicitly set), but if
so I could probably arrange to at least help with the changes.

Scott


_______________________________________________
ecf-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/ecf-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: etcd-provider - service keys to register service for discovery

Keimel, Christoph
Can I use something from the framework, the endpoint-id or something like that, or would I need to invent my own service configuration property?

-----Ursprüngliche Nachricht-----
Von: [hidden email] [mailto:[hidden email]] Im Auftrag von Scott Lewis
Gesendet: Samstag, 26. November 2016 08:12
An: [hidden email]
Betreff: Re: [ecf-dev] etcd-provider - service keys to register service for discovery

On 11/25/2016 12:20 PM, Keimel, Christoph wrote:

> Hi Scott,
>
> I've been playing with the etcd discovery provider lately with the aim
> of replacing zeroconf for our escape game platform. It's very nice and
> really, really (!) fast. In the escape game deployment scenario the
> system will be running longer than 20 minutes, which is why I set TTL
> to 0 so the services will not be removed automatically. (The default
> TTL is 1200 seconds.)
>
> This raises another issue though: As long as an application is shutdown gracefully the service key will be removed from etcd. But if you kill an application this is not the case. Unfortunately this is exactly what happens with the Raspberry Pi gateways on site. The guys power up the pi at the morning and at the evening they just kill the power to shut it down. The effect is that the keys will stay in etcd for ever. Since the keys are generated randomly for every session this leads to duplicates of services in etcd.
>
> I had a look at EtcdDiscoveryContainer and I was wondering if we could use a key to register the service with etcd that doesn't change between session? What do you think?

Yeah, I imagine that you could publish/advertise a service with a
consistent key.   I can't remember if this will require any changes to
the etcd provider (e.g. to allow the key to be explicitly set), but if so I could probably arrange to at least help with the changes.

Scott


_______________________________________________
ecf-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/ecf-dev
_______________________________________________
ecf-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/ecf-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: etcd-provider - service keys to register service for discovery

Scott Lewis-2
On 11/26/2016 12:29 AM, Keimel, Christoph wrote:
> Can I use something from the framework, the endpoint-id or something like that, or would I need to invent my own service configuration property?

Yes, there is an OSGi Remote Services Admin standard property:
endpoint.id.   If you set this property yourself (either via the service
properties passed to the registerService call, OR in the
overridingProperties passed to
RemoteServiceAdmin.exportService(svcRef,overridingProperties), then the
value you specify will be used for the endpoint.id in the description
published by discovery (via etcd in your case).  Take care though as
this endpoint.id must be globally unique...and it is usually a UUID
String although having UUID syntax is not required by RSA.

Scott


>
> -----Ursprüngliche Nachricht-----
> Von: [hidden email] [mailto:[hidden email]] Im Auftrag von Scott Lewis
> Gesendet: Samstag, 26. November 2016 08:12
> An: [hidden email]
> Betreff: Re: [ecf-dev] etcd-provider - service keys to register service for discovery
>
> On 11/25/2016 12:20 PM, Keimel, Christoph wrote:
>> Hi Scott,
>>
>> I've been playing with the etcd discovery provider lately with the aim
>> of replacing zeroconf for our escape game platform. It's very nice and
>> really, really (!) fast. In the escape game deployment scenario the
>> system will be running longer than 20 minutes, which is why I set TTL
>> to 0 so the services will not be removed automatically. (The default
>> TTL is 1200 seconds.)
>>
>> This raises another issue though: As long as an application is shutdown gracefully the service key will be removed from etcd. But if you kill an application this is not the case. Unfortunately this is exactly what happens with the Raspberry Pi gateways on site. The guys power up the pi at the morning and at the evening they just kill the power to shut it down. The effect is that the keys will stay in etcd for ever. Since the keys are generated randomly for every session this leads to duplicates of services in etcd.
>>
>> I had a look at EtcdDiscoveryContainer and I was wondering if we could use a key to register the service with etcd that doesn't change between session? What do you think?
> Yeah, I imagine that you could publish/advertise a service with a
> consistent key.   I can't remember if this will require any changes to
> the etcd provider (e.g. to allow the key to be explicitly set), but if so I could probably arrange to at least help with the changes.
>
> Scott
>
>
> _______________________________________________
> ecf-dev mailing list
> [hidden email]
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/ecf-dev
> _______________________________________________
> ecf-dev mailing list
> [hidden email]
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> https://dev.eclipse.org/mailman/listinfo/ecf-dev


_______________________________________________
ecf-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/ecf-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: etcd-provider - service keys to register service for discovery

Keimel, Christoph
Hi Scott,

thanks. I will prepare a pull request.

Since the endpoint.id is globally unique and is also generated randomly I think we could safly change the implementation to use this instead of EtcdServiceInfoKey. Or would you rather make this configurable with a system property?

i.e. org.eclipse.ecf.provider.etcd.keyformat=endpoint.id

Cheers,
Christoph

-----Ursprüngliche Nachricht-----
Von: [hidden email] [mailto:[hidden email]] Im Auftrag von Scott Lewis
Gesendet: Samstag, 26. November 2016 20:56
An: [hidden email]
Betreff: Re: [ecf-dev] etcd-provider - service keys to register service for discovery

On 11/26/2016 12:29 AM, Keimel, Christoph wrote:
> Can I use something from the framework, the endpoint-id or something like that, or would I need to invent my own service configuration property?

Yes, there is an OSGi Remote Services Admin standard property:
endpoint.id.   If you set this property yourself (either via the service
properties passed to the registerService call, OR in the overridingProperties passed to RemoteServiceAdmin.exportService(svcRef,overridingProperties), then the value you specify will be used for the endpoint.id in the description published by discovery (via etcd in your case).  Take care though as this endpoint.id must be globally unique...and it is usually a UUID String although having UUID syntax is not required by RSA.

Scott


>
> -----Ursprüngliche Nachricht-----
> Von: [hidden email] [mailto:[hidden email]] Im Auftrag von Scott Lewis
> Gesendet: Samstag, 26. November 2016 08:12
> An: [hidden email]
> Betreff: Re: [ecf-dev] etcd-provider - service keys to register service for discovery
>
> On 11/25/2016 12:20 PM, Keimel, Christoph wrote:
>> Hi Scott,
>>
>> I've been playing with the etcd discovery provider lately with the aim
>> of replacing zeroconf for our escape game platform. It's very nice and
>> really, really (!) fast. In the escape game deployment scenario the
>> system will be running longer than 20 minutes, which is why I set TTL
>> to 0 so the services will not be removed automatically. (The default
>> TTL is 1200 seconds.)
>>
>> This raises another issue though: As long as an application is shutdown gracefully the service key will be removed from etcd. But if you kill an application this is not the case. Unfortunately this is exactly what happens with the Raspberry Pi gateways on site. The guys power up the pi at the morning and at the evening they just kill the power to shut it down. The effect is that the keys will stay in etcd for ever. Since the keys are generated randomly for every session this leads to duplicates of services in etcd.
>>
>> I had a look at EtcdDiscoveryContainer and I was wondering if we could use a key to register the service with etcd that doesn't change between session? What do you think?
> Yeah, I imagine that you could publish/advertise a service with a
> consistent key.   I can't remember if this will require any changes to
> the etcd provider (e.g. to allow the key to be explicitly set), but if so I could probably arrange to at least help with the changes.
>
> Scott
>
>
> _______________________________________________
> ecf-dev mailing list
> [hidden email]
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/ecf-dev
> _______________________________________________
> ecf-dev mailing list
> [hidden email]
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> https://dev.eclipse.org/mailman/listinfo/ecf-dev


_______________________________________________
ecf-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/ecf-dev
_______________________________________________
ecf-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/ecf-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: etcd-provider - service keys to register service for discovery

Scott Lewis-2
Hi Christoph,

On 11/28/2016 1:22 PM, Keimel, Christoph wrote:
> Hi Scott,
>
> thanks. I will prepare a pull request.
>
> Since the endpoint.id is globally unique and is also generated randomly I think we could safly change the implementation to use this instead of EtcdServiceInfoKey. Or would you rather make this configurable with a system property?

I've just had a chance to look at this again after a year or so away
from it, and now I remember what I was doing with the EtcdServiceInfoKey.

The EtcdServiceInfoKey consists of two parts: <uuid1>.<uuid2>  The uuid1
is referred to in the code as a 'sessionId' associated with the
container upon creation (via the EtcdDiscoveryContainerConfig), and the
uuid2 part is the serviceInfo-specific uuid1.   Both uuid1 and uuid2
have UUID syntax.

Now, the sessionId part of the EtcdServiceInfoKey can be set for the
singleton EtcdDiscoveryContainer by setting this system property:

ecf.discovery.etcd.sessionid

e.g. something like this

-Decf.discovery.etcd.sessionid=4ed4bf64-42a7-48dc-abda-59a0d744a895

Then in EtcdDiscoveryContainer.registerService(IServiceInfo serviceInfo)
line 128 is currently this code:

EtcdServiceInfoKey siKey = new EtcdServiceInfoKey();

This constructor creates a full key with this syntax: <sessionId>.<new
random uuid>

There is already a second constructor:   EtcdServiceInfoKey(String
sessionId, String serviceInfoId) that can be used something like this:

                                // get the endpoint.id from the service
properties.  If set this will be the endpoint description UUID
line 128:        String endpointid =
serviceInfo.getServiceProperties().getPropertyString("endpoint.id");
//$NON-NLS-1$
line 129:        EtcdServiceInfoKey siKey = (endpointid == null)?new
EtcdServiceInfoKey():new EtcdServiceInfoKey(this.sessionId,endpointid);

What this does is use the UUID created for the endpoint.id (if present)
to set the EtcdServiceInfoKey to <sessionid>.<endpointid>, which is what
you want, I think, as the sessionId can be set via the system property
above, and the endpointid will be set to whatever the endpoint.id
is...and as discussed this can be set/fixed by setting the 'endpoint.id'
service property when the remote service is registered.   As long as the
sessionIds are unique for each remote service host, and the endpointids
are unique for each remote service instance then these should be
stored/updated with the same key in etcd server.

Does this make sense?   Will it work as necessary for your use case?  If
so I can add the code above to the
EtcdDiscoveryContainer.registerService method, but perhaps you want to
add it (along with system and service property setting) to test first?

Thanks,

Scott


>
> i.e. org.eclipse.ecf.provider.etcd.keyformat=endpoint.id
>
> Cheers,
> Christoph
>
> -----Ursprüngliche Nachricht-----
> Von: [hidden email] [mailto:[hidden email]] Im Auftrag von Scott Lewis
> Gesendet: Samstag, 26. November 2016 20:56
> An: [hidden email]
> Betreff: Re: [ecf-dev] etcd-provider - service keys to register service for discovery
>
> On 11/26/2016 12:29 AM, Keimel, Christoph wrote:
>> Can I use something from the framework, the endpoint-id or something like that, or would I need to invent my own service configuration property?
> Yes, there is an OSGi Remote Services Admin standard property:
> endpoint.id.   If you set this property yourself (either via the service
> properties passed to the registerService call, OR in the overridingProperties passed to RemoteServiceAdmin.exportService(svcRef,overridingProperties), then the value you specify will be used for the endpoint.id in the description published by discovery (via etcd in your case).  Take care though as this endpoint.id must be globally unique...and it is usually a UUID String although having UUID syntax is not required by RSA.
>
> Scott
>
>
>> -----Ursprüngliche Nachricht-----
>> Von: [hidden email] [mailto:[hidden email]] Im Auftrag von Scott Lewis
>> Gesendet: Samstag, 26. November 2016 08:12
>> An: [hidden email]
>> Betreff: Re: [ecf-dev] etcd-provider - service keys to register service for discovery
>>
>> On 11/25/2016 12:20 PM, Keimel, Christoph wrote:
>>> Hi Scott,
>>>
>>> I've been playing with the etcd discovery provider lately with the aim
>>> of replacing zeroconf for our escape game platform. It's very nice and
>>> really, really (!) fast. In the escape game deployment scenario the
>>> system will be running longer than 20 minutes, which is why I set TTL
>>> to 0 so the services will not be removed automatically. (The default
>>> TTL is 1200 seconds.)
>>>
>>> This raises another issue though: As long as an application is shutdown gracefully the service key will be removed from etcd. But if you kill an application this is not the case. Unfortunately this is exactly what happens with the Raspberry Pi gateways on site. The guys power up the pi at the morning and at the evening they just kill the power to shut it down. The effect is that the keys will stay in etcd for ever. Since the keys are generated randomly for every session this leads to duplicates of services in etcd.
>>>
>>> I had a look at EtcdDiscoveryContainer and I was wondering if we could use a key to register the service with etcd that doesn't change between session? What do you think?
>> Yeah, I imagine that you could publish/advertise a service with a
>> consistent key.   I can't remember if this will require any changes to
>> the etcd provider (e.g. to allow the key to be explicitly set), but if so I could probably arrange to at least help with the changes.
>>
>> Scott
>>
>>
>> _______________________________________________
>> ecf-dev mailing list
>> [hidden email]
>> To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/ecf-dev
>> _______________________________________________
>> ecf-dev mailing list
>> [hidden email]
>> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
>> https://dev.eclipse.org/mailman/listinfo/ecf-dev
>
> _______________________________________________
> ecf-dev mailing list
> [hidden email]
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> https://dev.eclipse.org/mailman/listinfo/ecf-dev
> _______________________________________________
> ecf-dev mailing list
> [hidden email]
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> https://dev.eclipse.org/mailman/listinfo/ecf-dev


_______________________________________________
ecf-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/ecf-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: etcd-provider - service keys to register service for discovery

Keimel, Christoph
Hi Scott,

I just got back to this and saw that you already made the change :-)
Thanks! When we meet beverage on me!

Cheers,
Christoph
________________________________________
Von: [hidden email] [[hidden email]]&quot; im Auftrag von &quot;Scott Lewis [[hidden email]]
Gesendet: Mittwoch, 30. November 2016 07:53
An: [hidden email]
Betreff: Re: [ecf-dev] etcd-provider - service keys to register service for discovery

Hi Christoph,

On 11/28/2016 1:22 PM, Keimel, Christoph wrote:
> Hi Scott,
>
> thanks. I will prepare a pull request.
>
> Since the endpoint.id is globally unique and is also generated randomly I think we could safly change the implementation to use this instead of EtcdServiceInfoKey. Or would you rather make this configurable with a system property?

I've just had a chance to look at this again after a year or so away
from it, and now I remember what I was doing with the EtcdServiceInfoKey.

The EtcdServiceInfoKey consists of two parts: <uuid1>.<uuid2>  The uuid1
is referred to in the code as a 'sessionId' associated with the
container upon creation (via the EtcdDiscoveryContainerConfig), and the
uuid2 part is the serviceInfo-specific uuid1.   Both uuid1 and uuid2
have UUID syntax.

Now, the sessionId part of the EtcdServiceInfoKey can be set for the
singleton EtcdDiscoveryContainer by setting this system property:

ecf.discovery.etcd.sessionid

e.g. something like this

-Decf.discovery.etcd.sessionid=4ed4bf64-42a7-48dc-abda-59a0d744a895

Then in EtcdDiscoveryContainer.registerService(IServiceInfo serviceInfo)
line 128 is currently this code:

EtcdServiceInfoKey siKey = new EtcdServiceInfoKey();

This constructor creates a full key with this syntax: <sessionId>.<new
random uuid>

There is already a second constructor:   EtcdServiceInfoKey(String
sessionId, String serviceInfoId) that can be used something like this:

                                // get the endpoint.id from the service
properties.  If set this will be the endpoint description UUID
line 128:        String endpointid =
serviceInfo.getServiceProperties().getPropertyString("endpoint.id");
//$NON-NLS-1$
line 129:        EtcdServiceInfoKey siKey = (endpointid == null)?new
EtcdServiceInfoKey():new EtcdServiceInfoKey(this.sessionId,endpointid);

What this does is use the UUID created for the endpoint.id (if present)
to set the EtcdServiceInfoKey to <sessionid>.<endpointid>, which is what
you want, I think, as the sessionId can be set via the system property
above, and the endpointid will be set to whatever the endpoint.id
is...and as discussed this can be set/fixed by setting the 'endpoint.id'
service property when the remote service is registered.   As long as the
sessionIds are unique for each remote service host, and the endpointids
are unique for each remote service instance then these should be
stored/updated with the same key in etcd server.

Does this make sense?   Will it work as necessary for your use case?  If
so I can add the code above to the
EtcdDiscoveryContainer.registerService method, but perhaps you want to
add it (along with system and service property setting) to test first?

Thanks,

Scott


>
> i.e. org.eclipse.ecf.provider.etcd.keyformat=endpoint.id
>
> Cheers,
> Christoph
>
> -----Ursprüngliche Nachricht-----
> Von: [hidden email] [mailto:[hidden email]] Im Auftrag von Scott Lewis
> Gesendet: Samstag, 26. November 2016 20:56
> An: [hidden email]
> Betreff: Re: [ecf-dev] etcd-provider - service keys to register service for discovery
>
> On 11/26/2016 12:29 AM, Keimel, Christoph wrote:
>> Can I use something from the framework, the endpoint-id or something like that, or would I need to invent my own service configuration property?
> Yes, there is an OSGi Remote Services Admin standard property:
> endpoint.id.   If you set this property yourself (either via the service
> properties passed to the registerService call, OR in the overridingProperties passed to RemoteServiceAdmin.exportService(svcRef,overridingProperties), then the value you specify will be used for the endpoint.id in the description published by discovery (via etcd in your case).  Take care though as this endpoint.id must be globally unique...and it is usually a UUID String although having UUID syntax is not required by RSA.
>
> Scott
>
>
>> -----Ursprüngliche Nachricht-----
>> Von: [hidden email] [mailto:[hidden email]] Im Auftrag von Scott Lewis
>> Gesendet: Samstag, 26. November 2016 08:12
>> An: [hidden email]
>> Betreff: Re: [ecf-dev] etcd-provider - service keys to register service for discovery
>>
>> On 11/25/2016 12:20 PM, Keimel, Christoph wrote:
>>> Hi Scott,
>>>
>>> I've been playing with the etcd discovery provider lately with the aim
>>> of replacing zeroconf for our escape game platform. It's very nice and
>>> really, really (!) fast. In the escape game deployment scenario the
>>> system will be running longer than 20 minutes, which is why I set TTL
>>> to 0 so the services will not be removed automatically. (The default
>>> TTL is 1200 seconds.)
>>>
>>> This raises another issue though: As long as an application is shutdown gracefully the service key will be removed from etcd. But if you kill an application this is not the case. Unfortunately this is exactly what happens with the Raspberry Pi gateways on site. The guys power up the pi at the morning and at the evening they just kill the power to shut it down. The effect is that the keys will stay in etcd for ever. Since the keys are generated randomly for every session this leads to duplicates of services in etcd.
>>>
>>> I had a look at EtcdDiscoveryContainer and I was wondering if we could use a key to register the service with etcd that doesn't change between session? What do you think?
>> Yeah, I imagine that you could publish/advertise a service with a
>> consistent key.   I can't remember if this will require any changes to
>> the etcd provider (e.g. to allow the key to be explicitly set), but if so I could probably arrange to at least help with the changes.
>>
>> Scott
>>
>>
>> _______________________________________________
>> ecf-dev mailing list
>> [hidden email]
>> To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/ecf-dev
>> _______________________________________________
>> ecf-dev mailing list
>> [hidden email]
>> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
>> https://dev.eclipse.org/mailman/listinfo/ecf-dev
>
> _______________________________________________
> ecf-dev mailing list
> [hidden email]
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> https://dev.eclipse.org/mailman/listinfo/ecf-dev
> _______________________________________________
> ecf-dev mailing list
> [hidden email]
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> https://dev.eclipse.org/mailman/listinfo/ecf-dev


_______________________________________________
ecf-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/ecf-dev
_______________________________________________
ecf-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/ecf-dev
Loading...